Escape ${foo} on strings when configuration is received from Elastic-Agent

[belimawr]

When the Elastic-Agent computes Unit Configs and send them to Beats, there might be some values in the format ${var} in the config, Beats tries to process them as environment variables and will fail to start the input if they cannot be resolved.

An example of where this can happen is when using Traeffik HTTP Middleware: https://doc.traefik.io/traefik/middlewares/http/redirectregex/.

The BeatV2Manager should escape all configurations received by the Elastic-Agent as they're ready to use and any variable replacement will be done by the Elastic-Agent, not the Beat.

For some reasoning/discussion about this solution, see the related issue.

Steps to reproduce

1. Deploy a container with the following docker-compose file

version: "3.7"

services:
  flog:
    image: mingrammer/flog
    labels:
        traefik: "http://mydomain/$${1}"
    command:
      - flog
      - "-d"
      - "1"
      - "-s"
      - "1"
      - "-l"

2. Deploy Elastic-Agent with Docker integration configured to collect logs

3. On the host run elastic-agent status --output=json

You'll see the error:

    "state": 3,
    "message": "1 or more components/units in a failed state",
    "components": [
        {
            "id": "filestream-default",
            "name": "filestream",
            "state": 2,
            "message": "Healthy: communicating with pid '46416'",
            "units": [
                {
                    "unit_id": "filestream-default",
                    "unit_type": 1,
                    "state": 4,
                    "message": "[failed to reloading inputs: 1 error: Unable to hash given config: missing field accessing 'processors']"
                },
                {
                    "unit_id": "filestream-default-filestream-docker-2c170cf0-df96-11ed-9bb4-83c1ad2564eb-docker-900d5d4110961c296621a1944c146b9a86ba3b190ea6e8403b3b0b7460c20a31",
                    "unit_type": 0,
                    "state": 4,
                    "message": "[failed to reloading inputs: 1 error: Unable to hash given config: missing field accessing 'processors']"
                }
            ],
            "version_info": {
                "name": "beat-v2-client",
                "version": "8.7.42",
                "meta": {
                    "build_time": "0001-01-01 00:00:00 +0000 UTC",
                    "commit": "unknown"
                }
            }
        },

4. If you inspect the container `docker inspect you'll see the label

            "Labels": {
                "traefik": "http://mydomain/${1}"
            }

Related issues

• Replaces Escape ${foo} on strings when computing a unit configuration  elastic-agent#2534

For confirmed bugs, please report:

• Version: 8.6.2
• Operating System: Linux, probably all OSes.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!