Context deadline exceeded while using slack notifier

[sathiyajith]

Im getting this error while trying to use --slack-wbehhook :-

time="2021-05-17T07:26:01Z" level=debug msg="reading config" annotations= debug=true dryRun=false excludedDaysOfYear="Apr1,Dec24" excludedPodNames="<nil>" excludedTimesOfDay= excludedWeekdays="Sat,Sun" gracePeriod=-1s includedPodNames="<nil>" interval=1m0s kinds="!DaemonSet" kubeconfig= labels= logFormat=text master= maxKill=1 metricsAddress=":8080" minimumAge=0s namespaceLabels= namespaces=chaoskube slackWebhook="https://hooks.slack.com/services/XXXXXXX" timezone=UTC
time="2021-05-17T07:26:01Z" level=info msg="starting up" dryRun=false interval=1m0s version=v0.21.0
time="2021-05-17T07:26:01Z" level=debug msg="using cluster config" kubeconfig= master=
W0517 07:26:01.873849       6 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2021-05-17T07:26:01Z" level=info msg="connected to cluster" master="https://10.96.0.1:443" serverVersion=v1.20.2
time="2021-05-17T07:26:01Z" level=info msg="setting pod filter" annotations= excludedPodNames="<nil>" includedPodNames="<nil>" kinds="!DaemonSet" labels= maxKill=1 minimumAge=0s namespaceLabels= namespaces=chaoskube
time="2021-05-17T07:26:01Z" level=info msg="setting quiet times" daysOfYear="[Apr 1 Dec24]" timesOfDay="[]" weekdays="[Saturday Sunday]"
time="2021-05-17T07:26:01Z" level=info msg="setting timezone" location=UTC name=UTC offset=0
time="2021-05-17T07:26:01Z" level=debug msg="found candidates" count=2
time="2021-05-17T07:26:01Z" level=debug msg="found victims" count=1
time="2021-05-17T07:26:01Z" level=info msg="terminating pod" name=nginx-deployment-556f98cd94-flwp9 namespace=chaoskube
time="2021-05-17T07:26:01Z" level=debug msg="calling deletePod endpoint" name=nginx-deployment-556f98cd94-flwp9 namespace=chaoskube terminator=DeletePod
time="2021-05-17T07:26:11Z" level=warning msg="failed to notify pod termination" err="1 error occurred:\n\t* Post \"https://hooks.slack.com/services/XXXXXXXXXXX\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n\n"
time="2021-05-17T07:26:11Z" level=debug msg=sleeping...

Can anyone please help.

[sathiyajith]

@linki, Can you please tell me how to give the Authentication Bearer token to push messages to slack?

[linki]

@sathiyajith I don't know if authentication was ever implemented. I don't see it in the code. It could be part of the webhook URL.

Maybe @GaruGaru knows more about it?

[sathiyajith]

@linki Yeah, I too am doubtful. Simple curl command like : curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' https://hooks.slack.com/services/XXXX seems to work fine. But, when I give the same webhook url here, I get this context deadline error. So, I thought we should give OAuth token as well in headers.

[GaruGaru]

@sathiyajith Your error context deadline exceeded (Client.Timeout exceeded while awaiting headers) doesn't look related to an incorrect authentication (which shouldn't be needed for webhooks), more likely you are experiencing a timeout on the http request.

That looks like some connectivity issue either by your side or server side problem to me.
Can you try to call the same url on the same instance where the chaoskube is running ?

[sathiyajith]

Okay. I tried the curl command in the same instance in which chaoskube is running. It is working.

[GaruGaru]

Are you running chaoskube inside kubernetes or on a vm ? Are the dns resolution and networking working correctly ? Do you have any NetworkPolicy that may be creating issues ?
I'll make some tests and i'll let you know

[sathiyajith]

Im running in kubernetes only. Networking is working fine. But I guess, this error is clearly because of something from my side kubernetes. Does cert-manager, MutatingWebhookConfiguration cause a problem? Because I have deployed them for other purpose. Can we please discuss this in any slack channel?

[GaruGaru]

Can confirm that everything is working correctly on my side. I don't know if there is any slack channel for this project ( @linki ).
To my understanding of this project those services should be interfere but @linki may know better.ì

[sathiyajith]

Okay, thanks. I will check from my side. I will close this issue too.