Graylog version 5.0.0 fails to index new messages using bulk API

[Ahmad-Faizan]

Describe the bug
Since this Graylog Helm chart does not support v5.0.0 yet, we deployed it alongside MongoDB 5.0.0 and OpenSearch 2.0.1 by overriding the image tag.
The cluster comes up online and we are able to log in to Graylog. The System > Overview also mentions that the cluster state is green. But as soon as we add an input to Graylog, and send a message, the pod logs throw indexing exceptions.

Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://opensearch-cluster-master:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying ( attempt #23). - {}

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-09T16:23:44Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.6", GitCommit:"6c23b67c202a4cfa7c76c3e1b370bd5f0e654f30", GitTreeState:"clean", BuildDate:"2022-11-09T17:13:23Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}

Which version of the chart: 2.2.0

What happened: Messages were not being indexed and throwing exceptions in the pod logs.

What you expected to happen: Messages should be displayed in the Graylog homepage

How to reproduce it (as minimally and precisely as possible):

1. Install MongoDB version 5.0.0
2. Install OpenSearch version 2.0.1 from this Helm chart
3. Install Graylog from this Helm chart
4. Change image tag to 5.0.0 in Graylog chart
5. Login to Graylog and check the cluster state is green or not under System > Overview
6. Add an input ( GELF TCP or Random Message Generator)
7. Check received messages under Search tab
8. The messages would not be displayed and the widget would have an error.
9. Check the pod logs of graylog pods, there will be an API indexing error.

Anything else we need to know:
A longer investigation has been added as a bug in the Graylog github issue
Graylog2/graylog2-server#14236

[Ahmad-Faizan]

The bug was due to manually changing the image tag to 5.0.0 but not modifying this line https://github.com/KongZ/charts/blob/main/charts/graylog/values.yaml#L403

Removing this line would be necessary when releasing this Helm chart with Graylog v5.0.0.