Azure Blob Storage via Azurite in CI runs

[VladLazar]

What & Why?

Currently we use the minio S3 container in order to emulate the S3 storage backend in cloud storage ducktape tests
(see tests/docker/docker-compose.yml).

In Q4 we will be introducing support for Azure Blob Storage (ABS), so we need an equivalent solution for running cloud storage ducktape tests with ABS. Microsoft provides an Azure emulator that can run in Docker: Azurite.

We should extend our test infrastructure such that we can run an Azurite container and have the test redpanda nodes connect to it. It should be configured to best resemble production Azure by using HTTPS and production style urls, although basic connectivity is the priority.

There are two ways in which ABS could be requested in a test run:

1. For the entire test suite: an env var (and a GitHub slash command) could be added to inform the test infra that ABS will be used for a given test run.
2. On a per test basis: tests could be parametrised to run against specific cloud providers

Option 2 provides more control and a permanent way of running Azure tests in CI without human intervention, but requires always running a minio and an Azurite container.

How?

Regardless of how we want to run these tests, the changes are similar. In broad strokes:

• Update the docker-compose configuration to include an Azurite container
• Make tests aware of the new endpoint by generalising redpanda.py::SISettings to work with both cloud storage solutions (the s3 specific globals S3_REGION, S3_BUCKET will need to be dealt with too).
• Generalise the bucket creation logic in redpanda.py::RedpandaService.start_si.
• (Specific to option 2): parametrise a few cloud storage tests to run against ABS too