Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Openshift] Pod keeps restarting as the registered watcher says "Config file /config/config.yaml is updated" #142

Closed
iamvvra opened this issue Aug 6, 2019 · 26 comments
Labels
Milestone

Comments

@iamvvra
Copy link

@iamvvra iamvvra commented Aug 6, 2019

1st, this project is good. I came here by chance before I wanted to write my own cluster event listener (kind of) to publish events directly to Mattermost.

Though I have deployed this bot in my Openshift cluster running on-premise, I have trouble running it.

The bug description
The deployed pod keeps restarting as the registerd watcher of the config/config.yaml says the file was modified. There is no sign of change from external source or manual.

To Reproduce
I use helm to create botkube in the botkube namespace. My Mattermost too is deployed in Openshift, but in a different namespace, mattermost.
helm install --name botkube --namespace botkube \ --set config.communications.mattermost.enabled=true \ --set config.communications.mattermost.url=http://mattermost.mattermost.svc:8065 \ --set config.communications.mattermost.token=generatedtoken \ --set config.communications.mattermost.team=my-team \ --set config.communications.mattermost.channel=cluster-stat \ --set config.settings.clustername=development-cluster \ --set config.settings.allowkubectl=true \ helm/botkube
(I have also used OCP specific route instead of svc).

Expected behavior
The bot should start with the provided configuration loaded in the /config/config.yaml.

Screenshots
Find a screenshot attached.
Screenshot 2019-08-06 at 7 30 29 PM

Let me know if you need any other detail. Also, tell me, please, if I am doing it right.

@iamvvra iamvvra added the bug label Aug 6, 2019
@iamvvra iamvvra changed the title Pod keeps restarts as the registered watcher says "Config file /config/config.yaml is updated" Pod keeps restarting as the registered watcher says "Config file /config/config.yaml is updated" Aug 6, 2019
@PrasadG193
Copy link
Member

@PrasadG193 PrasadG193 commented Aug 6, 2019

Hi @iamvvra ,
Are you using correct Mattermost URL? Because I see Mattermost is unreachable through the logs.

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 6, 2019

Yes, this log has that issue, but I fixed that by providing OCP route url - however, the initial messages reach Mattermost. See the screenshot
Screenshot 2019-08-06 at 8 05 01 PM

Every pod restart send the first message to Mattermost successfully.

@PrasadG193
Copy link
Member

@PrasadG193 PrasadG193 commented Aug 6, 2019

Are you still getting config update logs after setting OCP route url?

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 6, 2019

Yes, I get the same error even after I change it to a route, most recent deployment log for your ref. But, was that the cause of this issue?

Untitled 2

@PrasadG193
Copy link
Member

@PrasadG193 PrasadG193 commented Aug 6, 2019

That's strange. I hope you are using the latest release for deployment. Which cluster you are using? Also is there some process/job which might be updating the configmap? Could you please watch and tell me the status of configmap botkube-configmap?
Could you please post output of @BotKube version?

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 6, 2019

I am using the latest one v0.8.0. However, I tried to run this in my AWS EKS cluster with no restart (though the bot couldnt reach my Mattermost). There is no other jobs or process in this namespace to modify this configmap. As the pod keeps restarting I am attaching the event log.

Screenshot 2019-08-06 at 8 39 50 PM

@PrasadG193
Copy link
Member

@PrasadG193 PrasadG193 commented Aug 6, 2019

Could you please send logs of the BotKube pod? Also please make sure that you are using the latest helm chart from v0.8.0 branch while deploying

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 6, 2019

Log attached. I configured from your release branch source.

botkube.log

@PrasadG193
Copy link
Member

@PrasadG193 PrasadG193 commented Aug 6, 2019

ERRO[2019-08-06T16:03:31Z] Error in connecting to Mattermost team ngap
Error: : Invalid or expired session, please login again., 

Are you using correct token? Can you please try regenerating token?

@codenio
Copy link
Contributor

@codenio codenio commented Aug 7, 2019

Hi,
I guess there are some issues with mattermost latest version.

  • Got this same error, when tried to setup mattermost and botkube in my local (few weeks before).
    Checked by generating mutiple tokens(user and bot tokens), but session was expiring instantly.
  • got CORS error as well due to client4 model in mattermost-server go pkg. Try adding CORS settings in mattermost config file

So it is good to check with this prespective too, by trying with privous stable version of mattermost.
@iamvvra can you check the Debug logs by setting evn LOG_LEVEL=Debug in the botkube pod to debug this error further.

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 7, 2019

@aananthraj I am running the bot in log level set to debug, also, I have provided one of the log in this thread. Moreover, I got the same error having tried with previous v0.7.0 version.

Having tried with new token today, I get the same error Error connecting to Mattermost, additionally with Segmentation violation - error below.

INFO[2019-08-07T05:07:18Z] Config file /config/config.yaml is updated. Hence restarting the Pod
INFO[2019-08-07T05:07:18Z] Config file /config/config.yaml is updated. Hence restarting the Pod

ERRO[2019-08-07T05:07:18Z] Error in connecting to Mattermost team devs
Error: : Invalid or expired session, please login again.,
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1268edc]

goroutine 96 [running]:
github.com/infracloudio/botkube/pkg/bot.(*mmBot).Start.func1(0xc0006ea080, 0xc0002d9920)
/go/src/github.com/infracloudio/botkube/pkg/bot/mattermost.go:104 +0x5c
created by github.com/infracloudio/botkube/pkg/bot.(*mmBot).Start
/go/src/github.com/infracloudio/botkube/pkg/bot/mattermost.go:101 +0x5c4

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 7, 2019

An update, I find the connectivity to Mattermost is now established, however the pod continues to restart assuming the /config/config.yaml has changed. I will see in my side if any monitoring agents or tools provokes this false change alert.

Screenshot 2019-08-07 at 11 16 22 AM

@PrasadG193
Copy link
Member

@PrasadG193 PrasadG193 commented Aug 7, 2019

@iamvvra please don't use the older version of BotKube with newer helm charts. Which version of Mattermost you are using? We have tested it with Mattermost v5.11.1. We are trying to understand if the issue is with the latest version of Mattermost or from client side.

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 7, 2019

Screenshot 2019-08-07 at 11 21 07 AM

@PrasadG193
Copy link
Member

@PrasadG193 PrasadG193 commented Aug 7, 2019

Can you please try with v5.11.1+ version?

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 7, 2019

Sure, I'll try that.

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 7, 2019

I have deployed the said Mattermost 5.11.1 app separately. The BotKube reacts the same assuming the config.yaml was modified, keeps restarting the pod. However, the initial notification hits the Mattermost. I am not attaching logs as it was similar to one provided earlier.

Version & the channel the BotKube sends notifications. You can see multiple notifications everytime the pod restarts.

Screenshot 2019-08-07 at 8 57 18 PM

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 7, 2019

Is this have anything to do with Websockets?

@sanketsudake sanketsudake added this to the v0.9.0 milestone Aug 8, 2019
@PrasadG193
Copy link
Member

@PrasadG193 PrasadG193 commented Aug 8, 2019

@iamvvra I don't think it is a problem related to communication. Could you please verify that any other service is not updating botkube-configmap?

@cyberox
Copy link

@cyberox cyberox commented Aug 8, 2019

I'm having the same issue trying to connect Botkube 0.8.0 to slack on OpenShift.
I receive the same messages in slack, so the communication with slack is working. The pod keeps restarting because of the config.yaml change.

@PrasadG193
Copy link
Member

@PrasadG193 PrasadG193 commented Aug 8, 2019

I suspect if this is due to Kubernetes release. What version of K8s you are using? @cyberox @iamvvra

@cyberox
Copy link

@cyberox cyberox commented Aug 8, 2019

I'm running OpenShift 3.11, based on Kubernetes 1.11.

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 9, 2019

Screenshot 2019-08-09 at 12 58 53 PM

@PrasadG193 PrasadG193 changed the title Pod keeps restarting as the registered watcher says "Config file /config/config.yaml is updated" [Openshift] Pod keeps restarting as the registered watcher says "Config file /config/config.yaml is updated" Aug 9, 2019
@PrasadG193
Copy link
Member

@PrasadG193 PrasadG193 commented Aug 9, 2019

I tried reproducing the issue on K8s v1.11 - minikube, GKE and EKS, couldn't reproduce it. I think it has to do with how Openshift manages resources. I think it updates mounted files for some reason. For now, I am opening another issue to have an ability to disable config file watcher and disable automatic restart on config changed
Edit:
Opened issue: #150

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 9, 2019

Yes, a flag passed (helm too) on to the watcher to disable the auto reboot, default being true.

@iamvvra
Copy link
Author

@iamvvra iamvvra commented Aug 12, 2019

I have tested this fix, it works!

Screenshot 2019-08-12 at 3 11 39 PM

Screenshot 2019-08-12 at 3 10 30 PM

Screenshot 2019-08-12 at 3 10 53 PM

Screenshot 2019-08-12 at 3 11 17 PM

@PrasadG193 PrasadG193 closed this Sep 30, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants