New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tick missed to fire if Ready() is taking a long time #9939

Open
manishrjain opened this Issue Jul 19, 2018 · 0 comments

Comments

2 participants
@manishrjain
Copy link
Contributor

manishrjain commented Jul 19, 2018

In Dgraph, when we sometimes have to retrieve a snapshot, or if a server gets killed and rejoins the cluster, it takes a while to retrieve the snapshot, or replay the raft logs -- during this time, we get these warnings:

2018/07/19 20:57:18 node.go:400: WARN: A tick missed to fire. Node blocks too long!
2018/07/19 20:57:18 node.go:400: WARN: A tick missed to fire. Node blocks too long!
2018/07/19 20:57:18 node.go:400: WARN: A tick missed to fire. Node blocks too long!

Can we do something about them, and if so what can we do? Should we run ticks in a separate goroutine than Ready? Currently, we have them all in one select loop.

		case <-ticker.C:
			n.Raft().Tick()

		case rd := <-n.Raft().Ready():

@gyuho gyuho added the area/raft label Jul 20, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment