Chore: refactoring jsonnet to explicitly include the deployment mode in the config

[pracucci]

What this PR does

#3379 introduced read-write deployment mode support in jsonnet. When enabled, the read-write deployment mode jsonnet "uninstall" all microservices K8S resources (e.g. ingester statefulset). This was a trick introduced to not touch the microservices deployment at all in a initial stage.

In this PR I'm proposing a refactoring to introduce deployment_mode in the jsonnet config (defaults to microservices) and then use it when defining K8S resources (e.g. do not define ingester_statefulset at all if deployment_mode != 'microservices').

This PR is expected to be a no-op with regards to all jsonnet tests.

Which issue(s) this PR fixes or relates to

Fixes #3364

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[dimitarvdimitrov]

Not a jsonnet expert, but LGTM

[pracucci]

I will keep this PR open for at least 1 business day to see if anyone has any objection.

[colega]

LGTM.

One small nit is that I feel comparing to a string value can be a little bit fragile.

We could consider having $._config.is_microservices_deployment_mode and $._config.is_read_write_deployment_mode variables and referencing those, making the compiler enforce the correctness, and have a validation in the config to ensure that exactly one is set.

[pracucci]

LGTM.

One small nit is that I feel comparing to a string value can be a little bit fragile.

We could consider having $._config.is_microservices_deployment_mode and $._config.is_read_write_deployment_mode variables and referencing those, making the compiler enforce the correctness, and have a validation in the config to ensure that exactly one is set.

@colega Good feedback. WDYT about d3ed143 ?

[56quarters]

LGTM, one small suggestion.