Add integration test running Mimir in read-write deployment mode

[pracucci]

We should create an integration test to run Mimir in read-write deployment mode.

---

Cases / components to cover (see #3363 (comment)):

• ingest data and query from ingester (covers distributor, ingester, querier) - Read-write deployment mode integration test #3423
• ingest data and query from long-term storage (covers distributor, ingester, querier, store-gateway) - Read-write deployment mode integration tests: querying from long-term storage and alerting rules #3683
• recording rule (covers ruler) - Read-write deployment mode: add integration test for recording rule evaluation #3510
• alerting rule (covers alertmanager) - Read-write deployment mode integration tests: querying from long-term storage and alerting rules #3683
• compaction (covers compactor) - Read-write deployment mode integration tests: compactor #3799

[charleskorn]

I've got a first test up in #3423.

This first test just covers a very straightforward scenario of writing some data and then reading it back again.

I'm still getting to grips with the system and the implications of the different deployment modes, so I'm not sure what else would be valuable to test - I'm not sure what risks / concerns the different deployment modes introduce. For example, should we test that other functionality like the compactor, ruler or alert manager are working correctly? Are there other things that warrant an integration test?

[pracucci]

I'm still getting to grips with the system and the implications of the different deployment modes, so I'm not sure what else would be valuable to test - I'm not sure what risks / concerns the different deployment modes introduce. For example, should we test that other functionality like the compactor, ruler or alert manager are working correctly? Are there other things that warrant an integration test?

Good question! My take:

• Focus on the differences between microservices and read-write deployment mode
• The main difference is how components are packaged, so the most important thing to test for me is that components successfully start/run with -target=write/read/backend respectively
• For this reason, I wouldn't test "features" (e.g. replication factor 3 vs 1) but more the "deployment" (can querier talk to ingesters? can querier query historical data from store-gateway?)

To answer your question, as a follow up of #3423 I would additionally test:

• Store-gateway / compactor (keeping as simplest as possible)
• Ruler (e.g. evaluate a recording rule)

[charleskorn]

Once #3799 is merged, I think this is done - are there any other cases you think we need to cover @pracucci?

[pracucci]

Once #3799 is merged, I think this is done - are there any other cases you think we need to cover @pracucci?

No other cases we should cover. Thanks!