allow to set an env var to undefined, aka default

[bcoca]

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

config

[lilatomic]

Hey, maybe a silly question, but were you intending that users would set the envvar to literally 'AnsibleUndefined'? Or that they'd be able to get an actual instance of AnsibleUndefined into the environment variable?
I initially thought that it was the second of those, and I wanted to make sure that it played nice with the (coming) feature to create undefined variables directly. But I couldn't think of a way to set the envvar during config, which led me to believe that I was on the wrong track and overthinking it.

Anyhow, here's a patch with some tests I made while fiddling with this:

From 5dee184cf8987d4e8d7d8dd65ad885c4d0d6b79c Mon Sep 17 00:00:00 2001
From: Daniel Goldman <danielgoldman4@gmail.com>
Date: Sat, 2 Oct 2021 05:36:17 +0000
Subject: [PATCH] add tests for config.manager loading envvars

---
 test/units/config/test.yml        | 22 ++++++++++++++++++
 test/units/config/test_manager.py | 38 +++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+)

diff --git a/test/units/config/test.yml b/test/units/config/test.yml
index 384a055b2d..d1850362ef 100644
--- a/test/units/config/test.yml
+++ b/test/units/config/test.yml
@@ -28,6 +28,28 @@ config_entry_bool:
     <<: *entry
     type: bool
     default: False
+config_entry_bool_default_to_true:
+    name: test config
+    default: True
+    description:
+      - This does nothing, its for testing
+    env:
+      - name: BOOL_ENVVAR
+    ini:
+      - section: defaults
+        key: bool_key
+    type: bool
+config_entry_int:
+    name: test config
+    description:
+      - This does nothing, its for testing
+    type: int
+    default: 42
+    env:
+      - name: INT_ENVVAR
+    ini:
+      - section: defaults
+        key: int_key
 config_entry_list:
     <<: *entry
     type: list
diff --git a/test/units/config/test_manager.py b/test/units/config/test_manager.py
index a957e39749..314203411f 100644
--- a/test/units/config/test_manager.py
+++ b/test/units/config/test_manager.py
@@ -22,6 +22,8 @@ cfg_file2 = os.path.join(curdir, 'test2.cfg')
 expected_ini = {'CONFIG_FILE': Setting(name='CONFIG_FILE', value=cfg_file, origin='', type='string'),
                 'config_entry': Setting(name='config_entry', value=u'fromini', origin=cfg_file, type='string'),
                 'config_entry_bool': Setting(name='config_entry_bool', value=False, origin=cfg_file, type='bool'),
+                'config_entry_bool_default_to_true': Setting(name='config_entry_bool_default_to_true', value=True, origin='default', type='bool'),
+                'config_entry_int': Setting(name='config_entry_int', value=42, origin='default', type='int'),
                 'config_entry_list': Setting(name='config_entry_list', value=['fromini'], origin=cfg_file, type='list'),
                 'config_entry_deprecated': Setting(name='config_entry_deprecated', value=u'fromini', origin=cfg_file, type='string'),
                 'config_entry_multi': Setting(name='config_entry_multi', value=u'morefromini', origin=cfg_file, type='string'),
@@ -108,6 +110,42 @@ class TestConfigManager:
     def test_value_and_origin_from_alt_ini(self):
         assert self.manager.get_config_value_and_origin('config_entry', cfile=cfg_file2) == ('fromini2', cfg_file2)
 
+    @pytest.mark.parametrize(
+        ['env_value', 'expected_value', 'expected_origin'],
+        [
+            ('fromenviron', 'fromenviron', "env: %s" % 'ENVVAR'),
+            ('', '', "env: %s" % 'ENVVAR'),
+            ('AnsibleUndefined', 'DEFAULT', 'default'),
+        ])
+    def test_value_and_origin_from_env(self, monkeypatch, env_value, expected_value, expected_origin):
+        # monkeypatching the environment has to happen through the py3compat module
+        # since py3compat._TextEnviron holds a reference to the `os.environ` dictionary
+        # and so calling setattr on `os.environ` doesn't replace the reference held by py3compat
+        # we should be able to monkeypatch `os.environ` once py3compat is deprecated
+        from ansible.utils import py3compat
+        monkeypatch.setattr(py3compat.environ, '_raw_environ', {'ENVVAR': env_value})
+
+        assert ConfigManager(None, os.path.join(curdir, 'test.yml')).get_config_value_and_origin('config_entry') == (expected_value, expected_origin)
+
+    def test_value_and_origin_from_env_empty_and_wrong_type_defaults(self, monkeypatch):
+        from ansible.utils import py3compat
+        monkeypatch.setattr(py3compat.environ, '_raw_environ', {'INT_ENVVAR': ''})
+
+        assert ConfigManager(None, os.path.join(curdir, 'test.yml')).get_config_value_and_origin('config_entry_int') == (42, 'default')
+
+    @pytest.mark.parametrize(
+        ['env_key', 'config_key', 'expected_value'],
+        [
+            ('ENVVAR', 'config_entry_bool', False),
+            ('BOOL_ENVVAR', 'config_entry_bool_default_to_true', True),
+        ])
+    def test_value_and_origin_from_env_empty_and_bool(self, monkeypatch, env_key, config_key, expected_value):
+        """Test that the empty string is not cast into a bool and the default is fallen back on"""
+        from ansible.utils import py3compat
+        monkeypatch.setattr(py3compat.environ, '_raw_environ', {env_key: ''})
+
+        assert ConfigManager(None, os.path.join(curdir, 'test.yml')).get_config_value_and_origin(config_key) == (expected_value, 'default')
+
     def test_value_from_alt_ini(self):
         assert self.manager.get_config_value('config_entry', cfile=cfg_file2) == 'fromini2'
 
-- 
2.25.1

[bcoca]

it is meant to allow setting but not setting an env var, which should trigger a default value

[lilatomic]

I don't think I know what it means to do that. But it sounds like that's not what those test cases test for. Can you describe how I would do that? I can then fixup the test case.

Also, should we add to the docs about this feature?

[bcoca]

get_config_value_and_origin would return the 'next' value/origin in set (observing normal precedence), if none are set, 'default' it is.

[lilatomic]

Ahok, gotcha. I think I was confused by the 'AnsibleUndefined' on line 497. All that means is that, if the value happens to be AnsibleUndefined, we consider that a failure for the env lookup and keep going. In the more common case, if the value is None, we keep going. Like before.

And the second piece of modified code (at the end of the function) is just to catch-and-transform the ValueError which could arise from attempting to get the default if it's an empty envvar (happening inside the first try-catch).

I'll see about getting some tests in on the second part.

[lilatomic]

I think I found a failing test case: config values requesting Booleans will cast the emptystring to False, which will be accepted as a valid boolean. This will therefore not fall back on the default value. I've updated the git patch above with these tests (and other which pass).

[bcoca]

the reason i'm using a specific token is that 'empty string' is a valid way of saying 'false' in shell

[lilatomic]

Oh, gotcha, so we should expect the boolean thing to fail. So the recommended usage to force the default is to set the envvar to AnsibleUndefined. Setting the envvar to '' will sometimes work, but is also valid for booleans and strings, so it won't always reset to the default.

Should we include a doc fragment to that effect with this MR?

[ansibot]

The test ansible-test sanity --test ansible-doc [explain] failed with the error:

Command "ansible-doc -l --json -t filter ansible.builtin" returned exit status 5.
>>> Standard Error
ERROR: Unhandled exception when retrieving '_ANSIBLE_CONNECTION_PATH': 'NoneType' object has no attribute 'strip'

Traceback (most recent call last):
  File "/root/ansible/lib/ansible/config/manager.py", line 553, in get_config_value
    value, _drop = self.get_config_value_and_origin(config, cfile=cfile, plugin_type=plugin_type, plugin_name=plugin_name,
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                    keys=keys, variables=variables, direct=direct)
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/ansible/lib/ansible/config/manager.py", line 631, in get_config_value_and_origin
    if value.strip() == 'ansible_omit':
       ^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'strip'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/ansible/lib/ansible/cli/__init__.py", line 91, in <module>
    from ansible import constants as C
  File "/root/ansible/lib/ansible/constants.py", line 188, in <module>
    set_constant(setting, config.get_config_value(setting, variables=vars()))
                          ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/ansible/lib/ansible/config/manager.py", line 558, in get_config_value
    raise AnsibleError(f"Unhandled exception when retrieving {config!r}.") from ex
ansible.errors.AnsibleError: Unhandled exception when retrieving '_ANSIBLE_CONNECTION_PATH': 'NoneType' object has no attribute 'strip'

click here for bot help