Guard pod keeps restarting ~ every 8 hours

[Evalle]

Hi folks!
We use Guard with the LDAP authenticator, and on our clusters it keeps crashing ~ every 8 hours.

NAME                                     READY     STATUS    RESTARTS   AGE
guard-75658b5b8d-67tt7   1/1            Running   31                   8d

Here is the log from the guard pod:

kubectl logs -p -c guard guard-75658b5b8d-67tt7 -n kube-system
I0904 22:50:36.373564 1 logs.go:19] FLAG: --alsologtostderr="false"
I0904 22:50:36.376170 1 logs.go:19] FLAG: --analytics="false"
I0904 22:50:36.376198 1 logs.go:19] FLAG: --auth-providers="[ldap]"
I0904 22:50:36.376216 1 logs.go:19] FLAG: --azure.client-id=""
I0904 22:50:36.376253 1 logs.go:19] FLAG: --azure.client-secret=""
I0904 22:50:36.376265 1 logs.go:19] FLAG: --azure.tenant-id=""
I0904 22:50:36.376272 1 logs.go:19] FLAG: --azure.use-group-uid="true"
I0904 22:50:36.376378 1 logs.go:19] FLAG: --clock-check-interval="10m0s"
I0904 22:50:36.376484 1 logs.go:19] FLAG: --github.base-url=""
I0904 22:50:36.376566 1 logs.go:19] FLAG: --gitlab.base-url=""
I0904 22:50:36.376583 1 logs.go:19] FLAG: --google.admin-email=""
I0904 22:50:36.376589 1 logs.go:19] FLAG: --google.sa-json-file=""
I0904 22:50:36.376596 1 logs.go:19] FLAG: --help="false"
I0904 22:50:36.376643 1 logs.go:19] FLAG: --ldap.auth-choice="Simple"
I0904 22:50:36.376670 1 logs.go:19] FLAG: --ldap.bind-dn="<some_value>"
I0904 22:50:36.376680 1 logs.go:19] FLAG: --ldap.bind-password="<some_value>"
I0904 22:50:36.376697 1 logs.go:19] FLAG: --ldap.ca-cert-file=""
I0904 22:50:36.376704 1 logs.go:19] FLAG: --ldap.group-member-attribute="member"
I0904 22:50:36.376814 1 logs.go:19] FLAG: --ldap.group-name-attribute="cn"
I0904 22:50:36.376845 1 logs.go:19] FLAG: --ldap.group-search-dn=""
I0904 22:50:36.376853 1 logs.go:19] FLAG: --ldap.group-search-filter="(objectClass=groupOfNames)"
I0904 22:50:36.376862 1 logs.go:19] FLAG: --ldap.is-secure-ldap="false"
I0904 22:50:36.376868 1 logs.go:19] FLAG: --ldap.keytab-file=""
I0904 22:50:36.376875 1 logs.go:19] FLAG: --ldap.server-address="<some_value>"
I0904 22:50:36.376883 1 logs.go:19] FLAG: --ldap.server-port="389"
I0904 22:50:36.376889 1 logs.go:19] FLAG: --ldap.service-account=""
I0904 22:50:36.376897 1 logs.go:19] FLAG: --ldap.skip-tls-verification="true"
I0904 22:50:36.376903 1 logs.go:19] FLAG: --ldap.start-tls="true"
I0904 22:50:36.376912 1 logs.go:19] FLAG: --ldap.user-attribute="uid"
I0904 22:50:36.376919 1 logs.go:19] FLAG: --ldap.user-search-dn="<some_value>"
I0904 22:50:36.376926 1 logs.go:19] FLAG: --ldap.user-search-filter="(objectClass=person)"
I0904 22:50:36.376936 1 logs.go:19] FLAG: --log_backtrace_at=":0"
I0904 22:50:36.376944 1 logs.go:19] FLAG: --log_dir=""
I0904 22:50:36.376950 1 logs.go:19] FLAG: --logtostderr="false"
I0904 22:50:36.376958 1 logs.go:19] FLAG: --max-clock-skew="2m0s"
I0904 22:50:36.376967 1 logs.go:19] FLAG: --ntp-server="0.pool.ntp.org"
I0904 22:50:36.376973 1 logs.go:19] FLAG: --secure-addr=":8443"
I0904 22:50:36.376981 1 logs.go:19] FLAG: --stderrthreshold="0"
I0904 22:50:36.376988 1 logs.go:19] FLAG: --tls-ca-file="<some_value>"
I0904 22:50:36.376995 1 logs.go:19] FLAG: --tls-cert-file="<some_value>"
I0904 22:50:36.377002 1 logs.go:19] FLAG: --tls-private-key-file="<some_value>"
I0904 22:50:36.377899 1 logs.go:19] FLAG: --token-auth-file=""
I0904 22:50:36.377925 1 logs.go:19] FLAG: --v="3"
I0904 22:50:36.377939 1 logs.go:19] FLAG: --vmodule=""
F0905 02:30:41.418665 1 server.go:44] read udp 10.42.248.2:55378->69.164.198.192:123: i/o timeout
goroutine 21 [running]:
github.com/appscode/guard/vendor/github.com/golang/glog.stacks(0xc42059ee00, 0xc4205ac140, 0x68, 0xa0)
/go/src/github.com/appscode/guard/vendor/github.com/golang/glog/glog.go:766 +0xcf
github.com/appscode/guard/vendor/github.com/golang/glog.(*loggingT).output(0x24c8f20, 0xc400000003, 0xc42059d760, 0x240e1a0, 0x9, 0x2c, 0x0)
/go/src/github.com/appscode/guard/vendor/github.com/golang/glog/glog.go:717 +0x322
github.com/appscode/guard/vendor/github.com/golang/glog.(*loggingT).printDepth(0x24c8f20, 0xc400000003, 0x1, 0xc420514f98, 0x1, 0x1)
/go/src/github.com/appscode/guard/vendor/github.com/golang/glog/glog.go:646 +0x12a
github.com/appscode/guard/vendor/github.com/golang/glog.(*loggingT).print(0x24c8f20, 0xc400000003, 0xc420514f98, 0x1, 0x1)
/go/src/github.com/appscode/guard/vendor/github.com/golang/glog/glog.go:637 +0x5a
github.com/appscode/guard/vendor/github.com/golang/glog.Fatal(0xc420514f98, 0x1, 0x1)
/go/src/github.com/appscode/guard/vendor/github.com/golang/glog/glog.go:1125 +0x53
github.com/appscode/guard/server.Server.ListenAndServe.func1(0xc4204d0a00, 0xc420331490)
/go/src/github.com/appscode/guard/server/server.go:44 +0x103
created by github.com/appscode/guard/server.Server.ListenAndServe
/go/src/github.com/appscode/guard/server/server.go:41 +0xe1b

It's probably related to this block of code.

Note that I've tried to change the NTP server to our local NTP server and I had the same error.

[tamalsaha]

You can disable the timestamp check. You need to set --clock-check-interval=0 to disable.

ref: #143 (comment)

Let me know if this fixes the issue for you.

[Evalle]

thanks, @tamalsaha, with --clock-check-interval=0 Guard is running for 2 days without a problem:

kubectl get po -n kube-system -l app=guard
NAME                     READY     STATUS    RESTARTS   AGE
guard-76d85485ff-5zhp7   1/1       Running   0          2d