No slack message on end-testsuite-failed event

[michal-jagiello-tmpl]

Describe the bug
I have configured Slack notifications on start and end of testsuite:

[
    {
      "ChannelID": "C04U20FRM7Y",
      "selector": {},
      "testName": [],
      "testSuiteName": ["tnap-testsuite"],
      "events": [
        "start-testsuite",
        "end-testsuite-success",
        "end-testsuite-failed",
        "end-testsuite-aborted",
        "end-testsuite-timeout"
      ]
    }
  ]

I got message properly on start-testsuite event, but on failure I don't.
Start testsuite event log:

{"level":"info","ts":1680873484.2352679,"caller":"event/emitter.go:128","msg":"event published","id":"19286859-3fda-48f9-a959-525d0e337667","type":"start-testsuite","resource":"","resourceId":"","executionName":"ts-tnap-testsuite-3","executionId":"6430180c8b80a5d821a6a6bc","labels":"argocd.argoproj.io/instance=tnap-testkube ","topic":"events.all","error":null}
{"level":"info","ts":1680873484.235629,"caller":"scheduler/testsuite_scheduler.go:99","msg":"Running steps","test":"ts-tnap-testsuite-3"}
{"level":"info","ts":1680873484.2359557,"caller":"scheduler/testsuite_scheduler.go:246","msg":"Abortion check started","test":"ts-tnap-testsuite-3","timeout":0}
{"level":"info","ts":1680873484.23902,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"d6465da7-d1fd-4201-8f9d-8289d0e25a97","event":"start-testsuite"}
{"level":"info","ts":1680873484.23929,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"81970d74-9281-4c38-b79c-26056a693494","event":"start-testsuite"}
{"level":"info","ts":1680873484.239325,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"870a8fff-fc4b-422c-9e53-fe701171d86f","event":"start-testsuite"}
{"level":"info","ts":1680873484.2393725,"caller":"websocket/listener.go:63","msg":"notifying websocket","id":"769b5789-adde-4534-86f2-ada68183dbe2","event":"start-testsuite"}
{"level":"info","ts":1680873484.2399592,"caller":"event/emitter.go:174","msg":"listener notified","listen-on":["start-test","end-test-success","end-test-failed","end-test-aborted","end-test-timeout","start-testsuite","end-testsuite-success","end-testsuite-failed","end-testsuite-aborted","end-testsuite-timeout","created","deleted","updated"],"queue-group":"websocket.allevents","selector":"","metadata":{"clients":"[]","events":"[start-test end-test-success end-test-failed end-test-aborted end-test-timeout start-testsuite end-testsuite-success end-testsuite-failed end-testsuite-aborted end-testsuite-timeout created deleted updated]","name":"websocket.allevents","selector":""},"id":"19286859-3fda-48f9-a959-525d0e337667","type":"start-testsuite","resource":"","resourceId":"","executionName":"ts-tnap-testsuite-3","executionId":"6430180c8b80a5d821a6a6bc","labels":"argocd.argoproj.io/instance=tnap-testkube ","topic":"events.all"}
{"level":"info","ts":1680873484.2403705,"caller":"slack/slack.go:187","msg":"Execution changed","status":"running"}
{"level":"info","ts":1680873484.2407188,"caller":"scheduler/testsuite_scheduler.go:325","msg":"executing testvariablesmap[]request{tnap-testsuite-basic-cps tnap-testsuite 0 map[]   map[]   [] []  [] map[] map[] true   false false 0 []  <nil>  <nil>   [] [] 0xc004e46540}","type":"executeTest","testSuiteName":"tnap-testsuite","name":"run:/basic-cps"}
{"level":"info","ts":1680873484.2413287,"caller":"slack/slack.go:81","msg":"channels to send event to","channels":["C04U20FRM7Y"]}

End testsuite log

{"level":"info","ts":1680873724.0710351,"caller":"event/emitter.go:128","msg":"event published","id":"4f208fb9-bbc3-4397-b757-e95b3d1ededa","type":"end-testsuite-failed","resource":"","resourceId":"","executionName":"ts-tnap-testsuite-3","executionId":"6430180c8b80a5d821a6a6bc","labels":"argocd.argoproj.io/instance=tnap-testkube ","topic":"events.all","error":"nats: maximum payload exceeded"}

Looks like message is too big for that event

To Reproduce
Steps to reproduce the behavior:

1. Run testsuite when Slack notifications are configured for start and end testsuite on specific channel
2. Wait until testsuite failed
3. See error

Expected behavior
Message is sent to Slack channel

Version / Cluster

• Which testkube version? 1.10.304
• What Kubernetes cluster? (e.g. GKE, EKS, Openshift etc, local KinD, local Minikube) multiple nodes on local cloud
• What Kubernetes version? Major:"1", Minor:"23", GitVersion:"v1.23.16"

[nicufk]

Hey @michal-jagiello-tmpl, can you try to modify the nats.maxPayload in the helm charts and see if you can still reproduce it?

[michal-jagiello-tmpl]

I've modify and set the value on testkube helm chart (which I tested on my env) kubeshop/helm-charts#459

[michal-jagiello-tmpl]

and with 8GB I have no issue