Environment variable resolution is broken when SPLUNK_DOUBLE_DOLLAR_CONFIG_SOURCE_COMPATIBLE is false

[crobert-1]

Versions: v0.71.0+

This is a result of this change.. The current impact is shown by the TestIncompatibleExpandedDollarSignsViaEnvConfigSource integration test.

Failing config section:

          - action: add_label
            new_label: double-dollar
            new_value: $${env:AN_ENVVAR}-suffix

The config resolution logic attempts to resolve the environment variable successfully, but rather than setting a config map up with the key value of the environment variable name as key, and environment value as the value, it simply creates a string of the environment variable value. This results in a panic when attempting to start the collector because it's attempting to merge a map with a string.

Panic location here.

Possible solutions include:

1. replacing the wrapped config provided entirely and just using core's.
2. Modifying the Retrieve method's logic to properly handle environment variables, and plain strings to merge them into a map. We'd have to be pretty careful with the second option as the config resolution logic is pretty involved, so just modifying values would be pretty risky.

[crobert-1]

This line is what ends up converting the environment variable value to a string of its own value. This call could either be replaced, modified, or the result could be edited to be the proper format. This would be if we chose option 2 in the description.

This is the Retrieve method that properly gets the env variable value. This method has an equivalent for file sources, HTTP, etc. These would all have to be properly handled if the second option is chosen.

[rmfitzpatrick]

Resolved by 70b75c7