Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
818 lines (743 sloc) 36.4 KB
00:14 < ndevos> hey all, I'm looking for a description on the network protocol thats being used, is it documened somewhere (as not in the src)?
00:14 < ndevos> s/documened/documented/ even
00:16 < tjackson> you mean like here http://www.gluster.com/community/documentation/index.php/Translators/protocol
00:16 < glusterbot> <http://goo.gl/yixTz> (at www.gluster.com)
00:16 * ndevos checks
00:17 < ndevos> tjackson: nah, more details, as in what packets go over the wire
00:18 < JoeJulian> tjackson: I think that would work. I've never fully understood all the data that's in that dumpfile, despite asking several times for some sort of overview.
00:18 < y4m4> ndevos: packets a RPC
00:18 < y4m4> *are
00:18 < y4m4> so the regular marshalling, unmarshalling style
00:19 < ndevos> y4m4: sure, but i'd like to decode them with wireshark - I have a long flight tomorrow and would like to give it a go
00:19 < y4m4> the recent RPC change hasn't been documented on what goes on the wire, but if you want to figure out 'tcpdump' is the beginning
00:20 < y4m4> netstat -ntlp | grep glusterfs to get the ports active and tcpdump gives the wiresharkable details
00:20 < JoeJulian> I have an open enhancement request for wireshark to decode glusterfs traffic.
00:20 * y4m4 +1
00:20 < y4m4> FTW JoeJulian !
00:21 < ndevos> JoeJulian: yeah, I've seen that one, but there was no work done yet?
00:21 < JoeJulian> Not that I'm aware of.
00:22 < ndevos> okay, I have my gluster cluster on my laptop, wireshark sources, compiler and vim
00:22 < ndevos> anything else you can think of that I would need?
00:22 < y4m4> 'tcpdump'
00:22 < JoeJulian> Dr. Pepper and Red Vines.
00:22 < ndevos> oh, and the gluster.git repo of course
00:22 < y4m4> some Weed!
00:22 * ndevos is Dutsh
00:23 < ndevos> *dutch
00:23 < JoeJulian> No, y4m4, we want him to focus.
00:23 < y4m4> he is 'Dutch', who would want to glusterfs packets and wireshark them
00:24 < y4m4> that's all you need i guess
00:24 < ndevos> okay, thanks, lets see if I can focus while on the plane :)
Low-level communication:
Sending data is done with writev() whic writes a number of 'struct iovec' buffers. Each iovec has a pointer to a start-address and the number of bytes that need to be sent.
The data of the first 'struct iovec' is a uint32_t called fraghdr (from 'struct ioq' in rpc/rpc-transport/socket/src/socket.h).
47 /* Given the 4-byte fragment header, returns non-zero if this fragment
48 * is the last fragment for the RPC record being assemebled.
49 * RPC Record marking standard defines a 32 bit value as the fragment
50 * header with the MSB signifying whether the fragment is the last
51 * fragment for the record being asembled.
52 */
53 #define RPC_LASTFRAG(fraghdr) ((uint32_t)(fraghdr & 0x80000000U))
54
55 /* Given the 4-byte fragment header, extracts the bits that contain
56 * the fragment size.
57 */
58 #define RPC_FRAGSIZE(fraghdr) ((uint32_t)(fraghdr & 0x7fffffffU))
After the first 'struct iovec', the following 'struct iovec' are appended:
- 0..N number of msg->rpchdr
- 0..N number of msg->proghdr
- 0..N number of msg->progpayload
RPC-Record Fragment Header
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| size of the fragment |L|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
\ \
/ RPC-Data /
\ \
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Size of the Fragment: 31 bits (unsigned integer)
This is the size of the RPC-fragment, including the first 4 bytes.
L bit: 1 bit
Bit indicating that this is the (L)ast Fragment of the RPC-record.
RPC-Data
/* sending a handshake
*
* glusterd_peer_rpc_notify()
* case RPC_CLNT_CONNECT:
* xlators/mgmt/glusterd/src/glusterd-handshake.c:
* glusterd_peer_handshake()
* gf_dump_req req
* req.gfs_id = 0xcafe
*
* glusterd_submit_request()
* xlators/mgmt/glusterd/src/glusterd-utils.c:
* xdr_serialize_generic()
* proc() -> xdr_gf_dump_req()
* xdr_u_quad_t(objp->gfs_id)
*
*
* rpcsvc_dump(rpcsvc_request_t *req)
* - "struct rpcsvc_request" = rpcsvc_request_t
* build_prog_details()
* loop through req->svc->programs and set attributes
* xdr_serialize_dump_rsp()
* xdr_serialize_generic()
* xdr_gf_dump_rsp() -> set gfs_id (quadruple), op_ret (int), op_errno (int)
* xdr_gf_prog_detail() -> progname, prognum, progver,
* (in case there is a ->next, xdr_gf_prog_detail())
* rpcsvc_submit_generic()
* rpcsvc_record_build_record()
* rpcsvc_transport_submit()
* rpcsvc_submit_generic()
* rpcsvc_record_build_record()
* rpcsvc_fill_reply()
* rpcsvc_record_build_header()
* rpcsvc_transport_submit()
* rpc_transport_submit_reply()
* socket_submit_reply()
* __socket_ioq_new()
* __socket_ioq_churn_entry()
*
*
* callback:
* glusterd_peer_dump_version_cbk()
* xdr_to_generic()
* xdrmem_create(..., XDR_DECODE)
* xdr_gf_dump_rsp()
*
* server executes server_getspec()
* xdr_to_generic (req->msg[0], &args, (xdrproc_t)xdr_gf_getspec_req)
* 1475 if (!xdr_u_int (xdrs, &objp->flags))
* 1477 if (!xdr_string (xdrs, &objp->key, ~0))
*
*/
/* peer probe from cli/src/cli-rpc-ops.c:
* gf_cli3_1_probe() -> outgoing
* gf_cli3_1_probe_cbk() <- incoming
*
* gf1_cli_probe_req req = {0,};
* req.hostname = hostname to peer probe
* req.port = (default) CLI_GLUSTERD_PORT
*
* cli/src/cli-rpc-ops.c:
* 1715 ret = cli_cmd_submit (&req, frame, cli_rpc_prog,
* 1716 GLUSTER_CLI_PROBE, NULL,
* 1717 this, gf_cli3_1_probe_cbk,
* 1718 (xdrproc_t)xdr_gf1_cli_probe_req);
* cli/src/cli-rpc-ops.c -> cli_rpc_prog
* 4152 struct rpc_clnt_program cli_prog = {
* 4153 .progname = "Gluster CLI",
* 4154 .prognum = GLUSTER_CLI_PROGRAM,
* 4155 .progver = GLUSTER_CLI_VERSION,
* 4156 .numproc = GLUSTER_CLI_PROCCNT,
* 4157 .proctable = gluster_cli_actors,
* 4158 };
*
* rpc/rpc-lib/src/protocol-common.h:
* 235 #define GLUSTER_CLI_PROGRAM 1238463 (= 0x12e5bf)
* 236 #define GLUSTER_CLI_VERSION 1
*
*
* cli/src/cli-cmd.c:
* 376 ret = cli_submit_request (req, frame, prog,
* 377 procnum, NULL, this, cbkfn, xdrproc);
*
* cli/src/cli.c:
* 222 cli_submit_request (void *req, call_frame_t *frame,
* 223 rpc_clnt_prog_t *prog,
* 224 int procnum, struct iobref *iobref,
* 225 xlator_t *this, fop_cbk_fn_t cbkfn, xdrproc_t xdrproc)
* ...
* 237 xdr_size = xdr_sizeof (xdrproc, req);
* 238 iobuf = iobuf_get2 (this->ctx->iobuf_pool, xdr_size);
* ...
* 268 ret = rpc_clnt_submit (global_rpc, prog, procnum, cbkfn,
* 269 &iov, count,
* 270 NULL, 0, iobref, frame, NULL, 0, NULL, 0, NULL);
*
* rpc/rpc-lib/src/rpc-clnt.c:
* 1367 rpc_clnt_submit (struct rpc_clnt *rpc, rpc_clnt_prog_t *prog,
* 1368 int procnum, fop_cbk_fn_t cbkfn,
* 1369 struct iovec *proghdr, int proghdrcount,
* 1370 struct iovec *progpayload, int progpayloadcount,
* 1371 struct iobref *iobref, void *frame, struct iovec *rsphdr,
* 1372 int rsphdr_count, struct iovec *rsp_payload,
* 1373 int rsp_payload_count, struct iobref *rsp_iobref)
* ....
* 1379 rpc_transport_req_t req;
* ....
* 1406 callid = rpc_clnt_new_callid (rpc);
* 1407
* 1408 conn = &rpc->conn;
* 1409
* 1410 rpcreq->prog = prog;
* 1411 rpcreq->procnum = procnum;
* 1412 rpcreq->conn = conn;
* 1413 rpcreq->xid = callid;
* 1414 rpcreq->cbkfn = cbkfn;
* ....
* 1418 if (proghdr) {
* 1419 proglen += iov_length (proghdr, proghdrcount);
* 1420 }
* 1421
* 1422 if (progpayload) {
* 1423 proglen += iov_length (progpayload,
* 1424 progpayloadcount);
* 1425 }
* 1426
* 1427 request_iob = rpc_clnt_record (rpc, frame, prog,
* 1428 procnum, proglen,
* 1429 &rpchdr, callid);
* rpc/rpc-lib/src/rpc-clnt.c:
* 1253 struct iobuf *
* 1254 rpc_clnt_record (struct rpc_clnt *clnt, call_frame_t *call_frame,
* 1255 rpc_clnt_prog_t *prog,int procnum, size_t payload_len,
* 1256 struct iovec *rpchdr, uint64_t callid)
* ....
* 1285 request_iob = rpc_clnt_record_build_record (clnt, prog->prognum,
* 1286 prog->progver,
* 1287 procnum, payload_len,
* 1288 callid, &au,
* 1289 rpchdr);
*
* rpc/rpc-lib/src/rpc-clnt.c:
* 1193 rpc_clnt_record_build_record (struct rpc_clnt *clnt, int prognum, int progver,
* 1194 int procnum, size_t payload, uint64_t xid,
* 1195 struct auth_glusterfs_parms *au,
* 1196 struct iovec *recbuf)
* ....
* 1223 ret = rpc_clnt_fill_request (prognum, progver, procnum, payload, xid,
* 1224 au, &request, auth_data);
*
*
* 1113 int
* 1114 rpc_clnt_fill_request (int prognum, int progver, int procnum, int payload,
* 1115 uint64_t xid, struct auth_glusterfs_parms *au,
* 1116 struct rpc_msg *request, char *auth_data)
* ....
* 1126 request->rm_xid = xid;
* 1127 request->rm_direction = CALL;
* 1128
* 1129 request->rm_call.cb_rpcvers = 2;
* 1130 request->rm_call.cb_prog = prognum;
* 1131 request->rm_call.cb_vers = progver;
* 1132 request->rm_call.cb_proc = procnum;
* ....
* 1137 ret = xdr_serialize_glusterfs_auth (auth_data, au);
* ....
* 1143 request->rm_call.cb_cred.oa_flavor = AUTH_GLUSTERFS;
* 1144 request->rm_call.cb_cred.oa_base = auth_data;
* 1145 request->rm_call.cb_cred.oa_length = ret;
* 1146
* 1147 request->rm_call.cb_verf.oa_flavor = AUTH_NONE;
* 1148 request->rm_call.cb_verf.oa_base = NULL;
* 1149 request->rm_call.cb_verf.oa_length = 0;
*
*
*
* 1231 recordhdr = rpc_clnt_record_build_header (record, pagesize, &request,
* 1232 payload);
*
* rpc/rpc-lib/src/rpc-clnt.c:
* 1157 struct iovec
* 1158 rpc_clnt_record_build_header (char *recordstart, size_t rlen,
* 1159 struct rpc_msg *request, size_t payload)
* ....
* 1166 ret = rpc_request_to_xdr (request, recordstart, rlen, &requesthdr);
*
* rpc/rpc-lib/src/xdr-rpcclnt.c
* 76 rpc_request_to_xdr (struct rpc_msg *request, char *dest, size_t len,
* 77 struct iovec *dst)
* ..
* 86 xdrmem_create (&xdr, dest, len, XDR_ENCODE);
* 87 if (!xdr_callmsg (&xdr, request)) {
* 88 gf_log ("rpc", GF_LOG_WARNING, "failed to encode call msg");
*
*
* http://sourceware.org/git/?p=glibc.git;a=blob;f=sunrpc/rpc_cmsg.c#l63
* 41 bool_t
* 42 xdr_callmsg (XDR *xdrs, struct rpc_msg *cmsg)
* ..
* 63 (void) IXDR_PUT_LONG (buf, cmsg->rm_xid);
* 64 (void) IXDR_PUT_ENUM (buf, cmsg->rm_direction);
* ..
* 67 (void) IXDR_PUT_LONG (buf, cmsg->rm_call.cb_rpcvers);
* ..
* 70 (void) IXDR_PUT_LONG (buf, cmsg->rm_call.cb_prog);
* 71 (void) IXDR_PUT_LONG (buf, cmsg->rm_call.cb_vers);
* 72 (void) IXDR_PUT_LONG (buf, cmsg->rm_call.cb_proc);
* 73 oa = &cmsg->rm_call.cb_cred;
* 74 (void) IXDR_PUT_ENUM (buf, oa->oa_flavor);
* 75 (void) IXDR_PUT_INT32 (buf, oa->oa_length);
* 76 if (oa->oa_length)
* 77 {
* 78 memcpy ((caddr_t) buf, oa->oa_base, oa->oa_length);
* 79 buf = (int32_t *) ((char *) buf + RNDUP (oa->oa_length));
* 80 }
* 81 oa = &cmsg->rm_call.cb_verf;
* 82 (void) IXDR_PUT_ENUM (buf, oa->oa_flavor);
* 83 (void) IXDR_PUT_INT32 (buf, oa->oa_length);
* 84 if (oa->oa_length)
* 85 {
* 86 memcpy ((caddr_t) buf, oa->oa_base, oa->oa_length);
*
*
* 1126 request->rm_xid = xid;
* 1127 request->rm_direction = CALL;
* 1128
* 1129 request->rm_call.cb_rpcvers = 2;
* 1130 request->rm_call.cb_prog = prognum;
* 1131 request->rm_call.cb_vers = progver;
* 1132 request->rm_call.cb_proc = procnum;
* 1133
* ....
* 1137 ret = xdr_serialize_glusterfs_auth (auth_data, au);
* 1138 if (ret == -1) {
* 1139 gf_log ("rpc-clnt", GF_LOG_DEBUG, "cannot encode credentials");
* 1140 goto out;
* 1141 }
* 1142
* 1143 request->rm_call.cb_cred.oa_flavor = AUTH_GLUSTERFS;
* 1144 request->rm_call.cb_cred.oa_base = auth_data;
* 1145 request->rm_call.cb_cred.oa_length = ret;
* 1146
* 1147 request->rm_call.cb_verf.oa_flavor = AUTH_NONE;
* 1148 request->rm_call.cb_verf.oa_base = NULL;
* 1149 request->rm_call.cb_verf.oa_length = 0;
*
* ....
* 1436 iobref_add (iobref, request_iob);
* 1437
* 1438 req.msg.rpchdr = &rpchdr;
* 1439 req.msg.rpchdrcount = 1;
* 1440 req.msg.proghdr = proghdr;
* 1441 req.msg.proghdrcount = proghdrcount;
* 1442 req.msg.progpayload = progpayload;
* 1443 req.msg.progpayloadcount = progpayloadcount;
* 1444 req.msg.iobref = iobref;
* 1445
* 1446 req.rsp.rsphdr = rsphdr;
* 1447 req.rsp.rsphdr_count = rsphdr_count;
* 1448 req.rsp.rsp_payload = rsp_payload;
* 1449 req.rsp.rsp_payload_count = rsp_payload_count;
* 1450 req.rsp.rsp_iobref = rsp_iobref;
* 1451 req.rpc_req = rpcreq;
* ....
* 1455 if (conn->connected == 0) {
* 1456 ret = rpc_transport_connect (conn->trans,
* 1457 conn->config.remote_port);
* ....
* 1464 ret = rpc_transport_submit_request (rpc->conn.trans,
* 1465 &req);
* 1466 if (ret == -1) {
* 1467 gf_log (conn->trans->name, GF_LOG_WARNING,
* 1468 "failed to submit rpc-request "
* 1469 "(XID: 0x%ux Program: %s, ProgVers: %d, "
* 1470 "Proc: %d) to rpc-transport (%s)", rpcreq->xid,
* 1471 rpcreq->prog->progname, rpcreq->prog->progver,
* 1472 rpcreq->procnum, rpc->conn.trans->name);
*
* rpc/rpc-lib/src/rpc-transport.c:
* -> ops->submit_request = socket_submit_request
* 2312 entry = __socket_ioq_new (this, &req->msg);
* ....
* 2317 ret = __socket_ioq_churn_entry (this, entry);
*
* rpc/rpc-lib/src/rpc-transport.h:
* 113 struct rpc_transport_msg {
* 114 struct iovec *rpchdr;
* 115 int rpchdrcount;
* 116 struct iovec *proghdr;
* 117 int proghdrcount;
* 118 struct iovec *progpayload;
* 119 int progpayloadcount;
* 120 struct iobref *iobref;
* 121 };
* 122 typedef struct rpc_transport_msg rpc_transport_msg_t;
*
* rpc/rpc-transport/socket/src/socket.h:
* 97 struct ioq {
* 98 union {
* 99 struct list_head list;
* 100 struct {
* 101 struct ioq *next;
* 102 struct ioq *prev;
* 103 };
* 104 };
* 105
* 106 uint32_t fraghdr;
* 107 struct iovec vector[MAX_IOVEC];
* 108 int count;
* 109 struct iovec *pending_vector;
* 110 int pending_count;
* 111 struct iobref *iobref;
* 112 };
*
*
*
* rpc/rpc-transport/socket/src/socket.c:
* 518 struct ioq *
* 519 __socket_ioq_new (rpc_transport_t *this, rpc_transport_msg_t *msg)
* ...
* 521 struct ioq *entry = NULL;
* ...
* 532 count = msg->rpchdrcount + msg->proghdrcount + msg->progpayloadcount;
* ...
* 550 entry->vector[0].iov_base = (char *)&entry->fraghdr;
* 551 entry->vector[0].iov_len = sizeof (entry->fraghdr);
* 552 entry->count = 1;
*
*
* rpc/rpc-transport/socket/src/socket.c:
* 624 __socket_ioq_churn_entry (rpc_transport_t *this, struct ioq *entry)
* ...
* 628 ret = __socket_writev (this, entry->pending_vector,
* 629 entry->pending_count,
* 630 &entry->pending_vector,
* 631 &entry->pending_count);
*
* 258 __socket_writev (rpc_transport_t *this, struct iovec *vector, int count,
* 259 struct iovec **pending_vector, int *pending_count)
* ...
* 263 ret = __socket_rwv (this, vector, count,
* 264 pending_vector, pending_count, NULL, 1);
*
* 147 __socket_rwv (rpc_transport_t *this, struct iovec *vector, int count,
* 148 struct iovec **pending_vector, int *pending_count, size_t *bytes,
* 149 int write)
* ...
* 172 if (write) {
* 173 ret = writev (sock, opvector, opcount);
* ...
* 180 } else {
* 181 ret = readv (sock, opvector, opcount);
*
*
*/
XDR: External Data Representation Standard
- http://tools.ietf.org/html/rfc4506
Reading XDR Data:
- all components are multiples of 4 bytes
- variable length (like strings) are padded with 0-bytes at the end
- no type specification, this is up to intepretation by the reader
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RPC-Client Record Header |L|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
\ \
/ RPC-Client Frame /
\ \
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RPC-Client Record-0 |
| . |
| . |
| RPC-Client Record-N |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Peer Probe Capture, 140 bytes in total
Start a handshake
nodeA -> nodeB
Packet Hdr
L<--size--> <---xid---> <direction> <cb_rpcvers>
0000 80 00 00 88 00 00 00 01 00 00 00 00 00 00 00 02 ................
<-cb_prog-> <-cb_vers-> <-cb_proc-> <oa_flavor>
0010 07 5b b8 6d 00 00 00 01 00 00 00 01 00 00 00 05 .[.m............
<oa_length> <oa_base x 12
0020 00 00 00 58 00 00 00 00 00 00 00 00 00 00 00 00 ...X............
oa_base x 16
0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
oa_base x 16
0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
oa_base x 16
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
oa_base x 16
0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
oa_base x 12 > <----gfs_id-
0070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
----------gfs_id------------------->
0080 00 00 00 00 00 00 00 00 00 00 ca fe ............
L = 1
size = 0x000088
0x88 = 136 -> matches the number of bytes (header is 4 bytes)
xid: 1
direction: CALL=0 (REPLY=1)
cb_rpcvers: 0x2
cb_prog: 0x075bb86d = 123451501 = GLUSTER_DUMP_PROGRAM
cb_vers: 0x1 = GLUSTER_DUMP_VERSION
cb_proc: GF_DUMP_DUMP = 1
oa = &cmsg->rm_call.cb_cred
oa_flavor: 5 = AUTH_GLUSTERFS
oa_length: 0x58 = 88
oa = &cmsg->rm_call.cb_verf
oa_flavor: 5 = AUTH_GLUSTERFS
oa_length: 0x58 = 88
gfs_id: 0xcafe = 51966
nodeB -> nodeA
Implementation in rpc/rpc-lib/src/rpcsvc.c:rpcsvc_dump():
\ gf_dump_rsp rsp
\ build_prog_details(&rsp) /* writes all progname+procnum+procver to rsp->prog[] */
|
|- xdr_gf_dump_rsp() /* calculate size of the buffer needed */
|
\ xdr_serialize_dump_rsp(iov, rsp)
|
\ xdr_serialize_generic(iov, rsp, xdr_gf_dump_rsp)
|
\ xdr_gf_dump_rsp(&rsp)
|- xdr_u_quad_t(gfs_id)
|- xdr_int(op_ret)
|- xdr_int(op_errno)
\ xdr_pointer(rsp->prog)
.\ xdr_gf_prog_detail(rsp->prog[])
|| /* recurse through all rsp->prog */
||- xdr_string(progname)
||- xdr_u_quad_t(prognum)
||- xdr_u_quad_t(progver)
'/
/
L<--size--> <---xid---> <direction> <cb_rpcvers>
0000 80 00 01 1c 00 00 00 01 00 00 00 01 00 00 00 00 ................
<-cb_prog-> <-cb_vers-> <-cb_proc-> <oa_flavor>
0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
<oa_length> <--gfs_id-> <--op_ret-> <-op_errno>
0020 00 00 00 00 00 00 00 00 00 00 00 16 00 00 00 01 ................
<-str_len-> <------"GF-DUMP"------> <-prognum->
0030 00 00 00 07 47 46 2d 44 55 4d 50 00 00 00 00 00 ....GF-DUMP.....
prognum---> <-------progver-------> <?????????>
0040 07 5b b8 6d 00 00 00 00 00 00 00 01 00 00 00 01 .[.m............
<-str_len-> <-------------"GlusterD0.0.1"------
0050 00 00 00 0d 47 6c 75 73 74 65 72 44 30 2e 30 2e ....GlusterD0.0.
----------> <-------prognum-------> <---progver
0060 31 00 00 00 00 00 00 00 00 13 d0 01 00 00 00 00 1...............
progver---> <?????????> <-str_len-> <----------
0070 00 00 00 01 00 00 00 01 00 00 00 10 47 6c 75 73 ............Glus
----"GlusterD svc cli"------------>
0080 74 65 72 44 20 73 76 63 20 63 6c 69 00 00 00 00 terD svc cli....
0090 00 12 e5 bf 00 00 00 00 00 00 00 01 00 00 00 01 ................
00a0 00 00 00 11 47 6c 75 73 74 65 72 44 20 73 76 63 ....GlusterD svc
00b0 20 6d 67 6d 74 00 00 00 00 00 00 00 00 12 e5 a1 mgmt...........
00c0 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 0f ................
00d0 47 6c 75 73 74 65 72 20 50 6f 72 74 6d 61 70 00 Gluster Portmap.
00e0 00 00 00 00 02 08 ae c0 00 00 00 00 00 00 00 01 ................
00f0 00 00 00 01 00 00 00 13 47 6c 75 73 74 65 72 46 ........GlusterF
0100 53 20 48 61 6e 64 73 68 61 6b 65 00 00 00 00 00 S Handshake.....
0110 00 db b4 a9 00 00 00 00 00 00 00 01 00 00 00 00 ................
L: 1
size: 0x11c = 284
xid: 1
direction: 1 (reply)
cb_rpcvers: 0 (no callback)
cb_prog: 0 (no callback)
cb_vers: 0 (no callback)
cb_proc: 0 (no callback)
oa_flavor: 0
oa_length: 0
gfs_id (quadrup): 0
op_ret (int): 0x16 = 22 (Invalid argument)
op_errno (int): 0x1 (Operation not permitted)
progname (string): len=0x7=7 -> this is rounded to 8 (0-padded on the end)
GF-DUMP
prognum (quadruple): 0
progver (quadruple): 0x07 0x5b 0xb5 0x6d = 123450733
What would these be? It seems that it repeats
<?????????>
00 00 00 01
<?????????>
00 00 00 01
nodeA -> nodeB
Packet Hdr
L<--size--> <---xid---> <direction> <cb_rpcvers>
0000 80 00 00 d8 00 00 00 02 00 00 00 00 00 00 00 02 ................
<-cb_prog-> <-cb_vers-> <-cb_proc-> <oa_flavor>
0010 00 12 e5 a1 00 00 00 01 00 00 00 01 00 00 00 05 ................
<oa_length> <oa_base x 12
0020 00 00 00 58 00 00 00 00 00 00 00 00 00 00 00 00 ...X............
0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
oa_base-->
0070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0080 00 00 00 00 00 00 00 90 00 00 00 45 00 00 00 d5 ...........E....
0090 00 00 00 f2 00 00 00 51 00 00 00 98 00 00 00 46 .......Q.......F
00a0 00 00 00 3a 00 00 00 80 00 00 00 c1 00 00 00 9a ...:............
00b0 00 00 00 04 00 00 00 49 00 00 00 ae 00 00 00 11 .......I........
00c0 00 00 00 84 00 00 00 0d 31 37 32 2e 33 31 2e 31 ........172.31.1
00d0 32 32 2e 35 39 00 00 00 00 00 00 00 22.59.......
L: 1 (last packet)
size: 0xd8 = 216
xid: 2
direction: 0 (CALL)
cb_rpcvers: 2
cb_prog: 0x12 0xe5 0xa1
oa_flavor: 5 = AUTH_GLUSTERFS
oa_length: 0x58 = 88
nodeB -> nodeA
0000 80 00 00 78 00 00 00 02 00 00 00 01 00 00 00 00 ...x............
0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 de ................
0020 00 00 00 78 00 00 00 d0 00 00 00 20 00 00 00 19 ...x....... ....
0030 00 00 00 b2 00 00 00 4e 00 00 00 ed 00 00 00 ac .......N........
0040 00 00 00 df 00 00 00 f0 00 00 00 0c 00 00 00 7d ...............}
0050 00 00 00 15 00 00 00 2b 00 00 00 1a 00 00 00 0d .......+........
0060 31 37 32 2e 33 31 2e 31 32 32 2e 35 39 00 00 00 172.31.122.59...
0070 00 00 00 00 00 00 00 00 00 00 00 00 ............
nodeA -> nodeB
0000 80 00 00 f0 00 00 00 03 00 00 00 00 00 00 00 02 ................
0010 00 12 e5 a1 00 00 00 01 00 00 00 02 00 00 00 05 ................
0020 00 00 00 58 00 00 00 00 00 00 00 00 00 00 00 00 ...X............
0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0080 00 00 00 00 00 00 00 90 00 00 00 45 00 00 00 d5 ...........E....
0090 00 00 00 f2 00 00 00 51 00 00 00 98 00 00 00 46 .......Q.......F
00a0 00 00 00 3a 00 00 00 80 00 00 00 c1 00 00 00 9a ...:............
00b0 00 00 00 04 00 00 00 49 00 00 00 ae 00 00 00 11 .......I........
00c0 00 00 00 84 00 00 00 0d 31 37 32 2e 33 31 2e 31 ........172.31.1
00d0 32 32 2e 35 39 00 00 00 00 00 00 00 00 00 00 14 22.59...........
00e0 00 00 00 01 00 00 00 05 00 00 00 02 63 6f 75 6e ............coun
00f0 74 00 30 00 t.0.
NodeB -> nodeA
0000 80 00 00 88 00 00 00 01 00 00 00 00 00 00 00 02 ................
0010 07 5b b8 6d 00 00 00 01 00 00 00 01 00 00 00 05 .[.m............
0020 00 00 00 58 00 00 00 00 00 00 00 00 00 00 00 00 ...X............
0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0080 00 00 00 00 00 00 00 00 00 00 ca fe ............
NodeA -> nodeB
0000 80 00 01 1c 00 00 00 01 00 00 00 01 00 00 00 00 ................
0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0020 00 00 00 00 00 00 00 00 00 00 00 16 00 00 00 01 ................
0030 00 00 00 07 47 46 2d 44 55 4d 50 00 00 00 00 00 ....GF-DUMP.....
0040 07 5b b8 6d 00 00 00 00 00 00 00 01 00 00 00 01 .[.m............
0050 00 00 00 0d 47 6c 75 73 74 65 72 44 30 2e 30 2e ....GlusterD0.0.
0060 31 00 00 00 00 00 00 00 00 13 d0 01 00 00 00 00 1...............
0070 00 00 00 01 00 00 00 01 00 00 00 10 47 6c 75 73 ............Glus
0080 74 65 72 44 20 73 76 63 20 63 6c 69 00 00 00 00 terD svc cli....
0090 00 12 e5 bf 00 00 00 00 00 00 00 01 00 00 00 01 ................
00a0 00 00 00 11 47 6c 75 73 74 65 72 44 20 73 76 63 ....GlusterD svc
00b0 20 6d 67 6d 74 00 00 00 00 00 00 00 00 12 e5 a1 mgmt...........
00c0 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 0f ................
00d0 47 6c 75 73 74 65 72 20 50 6f 72 74 6d 61 70 00 Gluster Portmap.
00e0 00 00 00 00 02 08 ae c0 00 00 00 00 00 00 00 01 ................
00f0 00 00 00 01 00 00 00 13 47 6c 75 73 74 65 72 46 ........GlusterF
0100 53 20 48 61 6e 64 73 68 61 6b 65 00 00 00 00 00 S Handshake.....
0110 00 db b4 a9 00 00 00 00 00 00 00 01 00 00 00 00 ................
NodeB -> nodeA
0000 80 00 00 78 00 00 00 03 00 00 00 01 00 00 00 00 ...x............
0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 de ................
0020 00 00 00 78 00 00 00 d0 00 00 00 20 00 00 00 19 ...x....... ....
0030 00 00 00 b2 00 00 00 4e 00 00 00 ed 00 00 00 ac .......N........
0040 00 00 00 df 00 00 00 f0 00 00 00 0c 00 00 00 7d ...............}
0050 00 00 00 15 00 00 00 2b 00 00 00 1a 00 00 00 0d .......+........
0060 31 37 32 2e 33 31 2e 31 32 32 2e 35 39 00 00 00 172.31.122.59...
0070 00 00 00 00 00 00 00 00 00 00 00 00 ............
nodeB -> nodeA
0000 80 00 00 d8 00 00 00 02 00 00 00 00 00 00 00 02 ................
0010 00 12 e5 a1 00 00 00 01 00 00 00 01 00 00 00 05 ................
0020 00 00 00 58 00 00 00 00 00 00 00 00 00 00 00 00 ...X............
0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0080 00 00 00 00 00 00 00 de 00 00 00 78 00 00 00 d0 ...........x....
0090 00 00 00 20 00 00 00 19 00 00 00 b2 00 00 00 4e ... ...........N
00a0 00 00 00 ed 00 00 00 ac 00 00 00 df 00 00 00 f0 ................
00b0 00 00 00 0c 00 00 00 7d 00 00 00 15 00 00 00 2b .......}.......+
00c0 00 00 00 1a 00 00 00 0e 31 37 32 2e 33 31 2e 31 ........172.31.1
00d0 32 32 2e 31 30 32 00 00 00 00 00 00 22.102......
nodeA -> nodeB
0000 80 00 00 78 00 00 00 02 00 00 00 01 00 00 00 00 ...x............
0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90 ................
0020 00 00 00 45 00 00 00 d5 00 00 00 f2 00 00 00 51 ...E...........Q
0030 00 00 00 98 00 00 00 46 00 00 00 3a 00 00 00 80 .......F...:....
0040 00 00 00 c1 00 00 00 9a 00 00 00 04 00 00 00 49 ...............I
0050 00 00 00 ae 00 00 00 11 00 00 00 84 00 00 00 0e ................
0060 31 37 32 2e 33 31 2e 31 32 32 2e 31 30 32 00 00 172.31.122.102..
0070 00 00 00 00 00 00 00 00 00 00 00 00 ............
nodeB -> nodeA
0000 80 00 00 f0 00 00 00 03 00 00 00 00 00 00 00 02 ................
0010 00 12 e5 a1 00 00 00 01 00 00 00 02 00 00 00 05 ................
0020 00 00 00 58 00 00 00 00 00 00 00 00 00 00 00 00 ...X............
0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0080 00 00 00 00 00 00 00 de 00 00 00 78 00 00 00 d0 ...........x....
0090 00 00 00 20 00 00 00 19 00 00 00 b2 00 00 00 4e ... ...........N
00a0 00 00 00 ed 00 00 00 ac 00 00 00 df 00 00 00 f0 ................
00b0 00 00 00 0c 00 00 00 7d 00 00 00 15 00 00 00 2b .......}.......+
00c0 00 00 00 1a 00 00 00 0e 31 37 32 2e 33 31 2e 31 ........172.31.1
00d0 32 32 2e 31 30 32 00 00 00 00 00 00 00 00 00 14 22.102..........
00e0 00 00 00 01 00 00 00 05 00 00 00 02 63 6f 75 6e ............coun
00f0 74 00 30 00 t.0.
nodeA -> nodeB
0000 80 00 00 78 00 00 00 03 00 00 00 01 00 00 00 00 ...x............
0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90 ................
0020 00 00 00 45 00 00 00 d5 00 00 00 f2 00 00 00 51 ...E...........Q
0030 00 00 00 98 00 00 00 46 00 00 00 3a 00 00 00 80 .......F...:....
0040 00 00 00 c1 00 00 00 9a 00 00 00 04 00 00 00 49 ...............I
0050 00 00 00 ae 00 00 00 11 00 00 00 84 00 00 00 0e ................
0060 31 37 32 2e 33 31 2e 31 32 32 2e 31 30 32 00 00 172.31.122.102..
0070 00 00 00 00 00 00 00 00 00 00 00 00 ............
dict parsing
- 8 items:
<volfile-checksum
0000 76 6f 6c 66 69 6c 65 2d 63 68 65 63 6b 73 75 6d volfile-checksum
> <0 > <?????????????????????> < volfile-key
0010 00 30 00 00 00 00 0b 00 00 00 09 76 6f 6c 66 69 .0.........volfi
volfile-key > < /vol_dht >
0020 6c 65 2d 6b 65 79 00 2f 76 6f 6c 5f 64 68 74 00 le-key./vol_dht.
<?????????????????????> < client-version
0030 00 00 00 0e 00 00 00 06 63 6c 69 65 6e 74 2d 76 ........client-v
client-version > < 3.2.5 > <???????
0040 65 72 73 69 6f 6e 00 33 2e 32 2e 35 00 00 00 00 ersion.3.2.5....
?????????????> <process-uuid
0050 0c 00 00 00 47 70 72 6f 63 65 73 73 2d 75 75 69 ....Gprocess-uui
uuid> < node1.ssa.example.net ......
0060 64 00 6e 6f 64 65 31 2e 73 73 61 2e 65 78 61 6d d.node1.ssa.exam
0070 70 6c 65 2e 6e 65 74 2d 32 31 30 33 2d 32 30 31 ple.net-2103-201
0080 32 2f 30 31 2f 30 37 2d 31 35 3a 33 33 3a 33 32 2/01/07-15:33:32
0090 3a 38 33 35 33 33 30 2d 76 6f 6c 5f 64 68 74 2d :835330-vol_dht-
node1.ssa.example.net > <???????????????????
00a0 63 6c 69 65 6e 74 2d 31 00 00 00 00 0c 00 00 00 client-1........
?> < fops-version > <
00b0 08 66 6f 70 73 2d 76 65 72 73 69 6f 6e 00 31 32 .fops-version.12
1298437 > <?????????????????????> <tran
00c0 39 38 34 33 37 00 00 00 00 0e 00 00 00 07 74 72 98437.........tr
transport-type > <socket
00d0 61 6e 73 70 6f 72 74 2d 74 79 70 65 00 73 6f 63 ansport-type.soc
socket > <?????????????????????> <remote-sub
00e0 6b 65 74 00 00 00 00 10 00 00 00 12 72 65 6d 6f ket.........remo
> </export
00f0 74 65 2d 73 75 62 76 6f 6c 75 6d 65 00 2f 65 78 te-subvolume./ex
> <?
0100 70 6f 72 74 2f 62 72 69 63 6b 5f 64 68 74 00 00 port/brick_dht..
???????????????????> <remote-host
0110 00 00 0b 00 00 00 06 72 65 6d 6f 74 65 2d 68 6f .......remote-ho
> <node2 >
0120 73 74 00 6e 6f 64 65 32 00 00 00 00 st.node2....
volfile-checksum: 0
unknown: 00 00 00 0b 00 00 00 09
volfile-key: /vol_dht
unknown: 00 00 00 0e 00 00 00 06
client-version: 3.2.5
unknown: 00 00 00 0c 00 00 00 47
process-uuid: node1.ssa.example.net
unknown: 00 00 00 0c 00 00 00 08
fops-version: 1298437
unknown: 00 00 00 0e 00 00 00 07
transport-type: socket
unknown: 00 00 00 10 00 00 00 12
remote-subvolume: /export/brick_dht
unknown: 00 00 00 0b 00 00 00 06
remote-host: node2
unknown: 00 00 00 <-- RPC fill bytes?