Log 1894: Updating Loki Operand to 2.4.1

[Red-GV]

This PR does the following:

• Updates to Loki Operand to the 2.4.1 image
• Enables the WAL capabilities and disables the handover mechanism
• Adds the Index Gateway component
• Converts Queriers from a StatefulSet to a Deployment

The ingesters must remain as a statefulset in order to use WAL.

JIRA: https://issues.redhat.com/browse/LOG-1894

[periklis]

Awesome work! 🚀 A couple of small improvements and we are set to go:

1. Please vet that we using explicitly the new defaults if we don't override them in Loki config or limits config (https://github.com/grafana/loki/blob/main/docs/sources/upgrading/_index.md#change-of-some-default-limits-to-common-values)
2. Same for configuration defaults (https://github.com/grafana/loki/blob/main/docs/sources/upgrading/_index.md#change-of-configuration-defaults)
3. Although we use the FIFO cache since the early beginning please vet that the configuration makes explicit usage of its configuration params (https://github.com/grafana/loki/blob/main/docs/sources/upgrading/_index.md#in-memory-fifo-caches-enabled-by-default)
4. Same for the ingester lifecycle final_sleep param (https://github.com/grafana/loki/blob/main/docs/sources/upgrading/_index.md#ingester-lifecycler-final_sleep-now-defaults-to-0s)
5. Same for memberlist https://github.com/grafana/loki/blob/main/docs/sources/upgrading/_index.md#memberlist-config-now-automatically-applies-to-all-non-configured-rings
6. Same for GRPC server settings https://github.com/grafana/loki/blob/main/docs/sources/upgrading/_index.md#changed-defaults-for-some-grpc-server-settings

Making the above explicit in the loki configuration enables faster lookup while troubleshooting.

[periklis]

LGTM