Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep getting "rafthttp: the clock difference against peer 5bd0394181f126dd is too high [10.449391606s > 1s]" #7051

Closed
heyrakeshsingh opened this issue Dec 22, 2016 · 5 comments

Comments

@heyrakeshsingh
Copy link

heyrakeshsingh commented Dec 22, 2016

I have setup 5node+2 proxy cluster (each on node on separate machine- 12 core, 32 GB RAM ) with db size 4gb. Running 5 clients where each client spawning 5 threads. The operation performed is

Put
update (put on same keys)
get (on these same keys)

The operation is performed for 1.6 M keys from each client (ie total 25 threads sending request to etcd cluster server for 8 M keys)

I am getting below logs:-

2016-12-22 17:00:29.475936 W | rafthttp: the clock difference against peer 5bd0394181f126dd is too high [10.449391606s > 1s]
2016-12-22 17:00:59.476151 W | rafthttp: the clock difference against peer 5bd0394181f126dd is too high [10.449443811s > 1s]
2016-12-22 17:01:29.476315 W | rafthttp: the clock difference against peer 5bd0394181f126dd is too high [10.449597395s > 1s]
2016-12-22 17:01:59.476514 W | rafthttp: the clock difference against peer 5bd0394181f126dd is too high [10.449581152s > 1s]

@xiang90
Copy link
Contributor

xiang90 commented Dec 22, 2016

The system clock is different on you machines. Enable NTP can fix this. Or you have to manually sync the clocks on different machines.

@xiang90 xiang90 closed this as completed Dec 22, 2016
@heyrakeshsingh
Copy link
Author

heyrakeshsingh commented Dec 23, 2016

@xiang90 The system date/time differences is in seconds but still it is coming and i observe this only while running with large data as i have mentioned. Will it impact performance..? its taking 31m for the fastest client for putting 1.6M data, (1.6M * 5 clients with 5 conncurrent ops are happening continuously). Also i observe snapshot and purge happening too frequently.

@heyitsanthony
Copy link
Contributor

@heyrakeshsingh the time difference will cause v2 TTLs to expire too soon / too late. I don't believe it will make a performance difference so long as there's no TTL keys.

@ghost
Copy link

ghost commented Mar 6, 2019

I am observing the same issue but though i have NTP installed on all Master vms and all have same time. any hint> am runing k8s 1.10.9

2019-03-05 19:22:56.134642 W | rafthttp: the clock difference against peer 6c8a781291acb5f2 is too high [33.951034466s > 1s]
2019-03-05 19:23:04.554709 I | mvcc: store.index: compact 199600
2019-03-05 19:23:04.554869 I | mvcc: finished scheduled compaction at 199600 (took 47.317µs)
2019-03-05 19:23:26.134821 W | rafthttp: the clock difference against peer 6c8a781291acb5f2 is too high [33.951289195s > 1s]

@krishnakanthati
Copy link

I am observing the same issue but though i have NTP installed on all Master vms and all have same time. any hint> am runing k8s 1.10.9

2019-03-05 19:22:56.134642 W | rafthttp: the clock difference against peer 6c8a781291acb5f2 is too high [33.951034466s > 1s] 2019-03-05 19:23:04.554709 I | mvcc: store.index: compact 199600 2019-03-05 19:23:04.554869 I | mvcc: finished scheduled compaction at 199600 (took 47.317µs) 2019-03-05 19:23:26.134821 W | rafthttp: the clock difference against peer 6c8a781291acb5f2 is too high [33.951289195s > 1s]

Did you restart ETCD after installing NTP?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

4 participants