Skip to content
This repository was archived by the owner on Mar 26, 2020. It is now read-only.

Handle peer connects/disconnects in store#589

Merged
kshlm merged 3 commits intogluster:masterfrom
prashanthpai:node-disconnect-event
Mar 1, 2018
Merged

Handle peer connects/disconnects in store#589
kshlm merged 3 commits intogluster:masterfrom
prashanthpai:node-disconnect-event

Conversation

@prashanthpai
Copy link
Copy Markdown
Contributor

Watch for liveness of nodes in store:
Log and emit local event every time a node connects to or disconnects
from the store.

Explicitly delete liveness key when an instance shuts down:
The default etcd session TTL is 30 seconds which is quite high.
When a glusterd2 instance is shutdown, it takes a while for other
nodes to notice this as TTL for the liveness key should expire. When
a glusterd2 instance is shutdown gracefully (SIGTERM), it can simply
delete its liveness key from the store and hence other nodes can see
this instantly.

This was the only option that has hyphen in them and I got it wrong
many times when I manually edited conf file to add it.

I iften rely on using external etcd during development. Elasticetcd
doesn't work on my machine most of the times.

Signed-off-by: Prashanth Pai <ppai@redhat.com>
The default etcd session TTL is 30 seconds which is quite high. When a
glusterd2 instance is shutdown, it takes a while for other nodes to
notice this as TTL for the liveness key should expire.

When a glusterd2 instance is shutdown gracefully (SIGTERM), it can
simply delete its liveness key from the store and hence other nodes can
see this instantly.

Signed-off-by: Prashanth Pai <ppai@redhat.com>
Watch for liveness of nodes in store. Log and emit local event every
time a node connects or disconnects from the store.

Signed-off-by: Prashanth Pai <ppai@redhat.com>
@prashanthpai prashanthpai force-pushed the node-disconnect-event branch from 1dda5a8 to 0c2c887 Compare March 1, 2018 07:20
@prashanthpai
Copy link
Copy Markdown
Contributor Author

Sample (log):

INFO[2018-03-01 12:39:24.716217] peer connected to store                       id=14abaa12-3170-43a6-b332-69a54a5b8d83 source="[liveness.go:48:events.(*livenessWatcher).Watch]"
INFO[2018-03-01 12:39:52.834964] peer disconnected from store                  id=14abaa12-3170-43a6-b332-69a54a5b8d83 source="[liveness.go:51:events.(*livenessWatcher).Watch]"

Sample (events.log):

$ cat /tmp/gd2_func_test/w{1..2}/log/events.log  | grep peer
{"id":"c2489a1b-49f4-4d84-bea2-f10c0e425a06","name":"peer.connected.store","data":{"peer.id":"14abaa12-3170-43a6-b332-69a54a5b8d83"},"origin":"22b65ca8-005a-4ce7-b4e3-1ead397b4e33","timestamp":"2018-03-01T12:39:24.716254801+05:30"}
{"id":"761dcb18-0eb0-4585-b5e7-2911aed0284d","name":"peer.disconnected.store","data":{"peer.id":"14abaa12-3170-43a6-b332-69a54a5b8d83"},"origin":"22b65ca8-005a-4ce7-b4e3-1ead397b4e33","timestamp":"2018-03-01T12:39:52.835159879+05:30"}

@prashanthpai prashanthpai requested review from aravindavk and kshlm March 1, 2018 07:29
Copy link
Copy Markdown
Member

@kshlm kshlm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@prashanthpai Can you check why the CI failed?

@kshlm
Copy link
Copy Markdown
Member

kshlm commented Mar 1, 2018

retest this please

@kshlm
Copy link
Copy Markdown
Member

kshlm commented Mar 1, 2018

The CI failures here and elsewhere is caused by the change https://review.gluster.org/19466 which introduced a new field in the options struct, but added it into the middle. This causes GD2 to segfault when it tries to read options when running against builds that contain this change. The CI uses nightly-master builds, which has this change.

@kshlm
Copy link
Copy Markdown
Member

kshlm commented Mar 1, 2018

I've temporarily removed the centos-ci requirement until https://review.gluster.org/19652 is merged and becomes available in nightly builds.

I've manually run the verification, and it passes. Merging.

@kshlm kshlm merged commit 9f08c39 into gluster:master Mar 1, 2018
@kshlm kshlm mentioned this pull request Mar 1, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants