Add option to not import host variables in to AWX when sourcing inventory from SCM

[grahamneville]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

• API
• UI

SUMMARY

Add option to not import host variables in to AWX when sourcing inventory from SCM

ENVIRONMENT

• AWX version: 1.0.3.38
• AWX install method: docker on linux
• Ansible version: 2.4.0.3

STEPS TO REPRODUCE

I would like the option for AWX to not import any varibles from SCM so when Ansible is run it continues to use variables inside the project folder.

[wenottingham]

What is the reason you would like it to behave that way?

[grahamneville]

1. The AWX variables seem to override that of what is in the projects folder.
2. Due to 1, I'm unable to alter variables in a new branch and run a playbook against an existing inventory sourced from SCM as varibles get overriden to pick up the variables imported by AWX. Meaning I have to alter my inventory SCM source to pick up variables from the new branch. A way around this, although I'd rather not, is to have multiple copies of my inventories with different names to pick up different variables from different branches.
3. I ran in to the same issue as reported in Integer key inside dictionary imported as string when inventory is sourced from a project #991. This meant I had to change a number of Jinja snippets in order to get my templates working again. This is an example where AWX is changing the behaviour of Ansible core.

[wenottingham]

cc @AlanCoding

Re: #991 - that is a different issue, we would not solve it this way.

For the first item... can you provide more detail on the setup/structure of your inventory and playbook repo?

[AlanCoding]

This is an example where AWX is changing the behaviour of Ansible core.

I poked at this recently, and I would claim that the behavior reported in #991 has nothing to do with inventory updates.

The problem is that we run inventory through a JSON pipeline. The standard does not support certain types in certain places (like integer keys). Even if you assumed that we solved this in inventory updates (which would be a very backwards thing to do first), then the playbooks still won't be using the accurate types.

This meant I had to change a number of Jinja snippets in order to get my templates working again.

It sounds like you might have discovered this already. Even if you entered in the types you wanted into the UI manually as YAML, I don't expect they would be reflected in your playbook correctly. Because, again, using JSON carries with it fundamental data type constraints.

Now, there might be some interesting cases where vars are defined in both the inventory folder and the playbook folder, and I would certainly test and triage those vars precedence issues if we knew exactly what we were looking at.

[pilou-]

@AlanCoding with AWX, variables defined in inventory folders (from a project) become dynamic inventory variables due to ansible-inventory usage: then precedence of these variables differs depending if ansible-playbook is directly executed (without AWX) or if AWX is used.

Using AWX and static inventory (from a project), it should be possible to bypass ansible-inventory processing.

[AlanCoding]

The extra layer of ansible-inventory should ideally not change the playbook behavior. In a nutshell, that is the entire reason for ansible-inventory to exist.

Using AWX and static inventory (from a project), it should be possible to bypass ansible-inventory processing.

This is just now how AWX is designed. You could possibly hack your config to point the inventory to some file in your project source (for the playbook), but doing so would break the linkage between the Ansible host & the DB host, and this will break a lot, because it was not designed to work this way.

Maybe you could add in a vars manager plugin (inside the project source), so that the vars are altered instead of the inventory content itself.

[grahamneville]

Any further thoughts on allowing this as a simple check box when creating inventories?

I'm trying to avoid the need to create multiple inventories each time I may need to do some development testing where I'm changing variables belonging to hosts (either via ini files or yaml files e.g. inventory/SITE/group_vars/all.yaml)

If I could just import the list of hosts and groups they belong to without their variables I don't need to duplicate inventories.

This would also work well with the new prompt for project feature that was added in AWX 7.0.0

[AlanCoding]

^ You should be able to use variables sourced from <playbook dir>/group_vars/all.yml, which may be different in various project branches used in different jobs. As your use case is currently worded, it seems like this should probably address it. Variables defined in these folders relative to the playbook should take higher precedence than the inventory variables.

[grahamneville]

So I'm currently using the Alternative dir layout as per the Ansible best practices documentation https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html

I'm not sure that even if I moved my group vars from inventory/<SITE>/group_vars/<GROUP>.yml to <playbook_dir>/groups_vars/<GROUP>.yml my problem would be solved.

I'd rather not do this anyway.

Is it a lot of work not to stop the import of vars?

[AlanCoding]

Why do you think you need this? I have so-far failed to construct a story in my head where this makes sense as a solution to a problem.

If you need different variables for various hosts for jobs using different source trees, then you have playbook adjacent variables. If you don't, then that's fine.

But I'm reading that you're not trying to apply different playbook-source-dependent variables but for some reason, you want host and group vars from the inventory content gone. That's not an end goal, and I don't see any end goal that it would aid in achieving.

[ghost]

I ran into a similar, maybe same, issue as OP. When running a template and overriding the SCM Branch the inventory variables do not get updated with the new branch for that run.

For example say production is using the "master" branch. The project is setup to use the master branch but allow for the branch to be overridden (for testing). Then the inventory is setup to use the SCM project for its source. This pulls in the projects variables from the master branch and sets them in AWX. Now if in a template I override the SCM branch to say "develop-me" the inventory variables of the master branch get merged with the project variables of the "develop-me" branch. This is a major issue that can cause unintended behavior due to this merge.

In my case the "develop-me" branch was testing a fix to issues caused by an incorrect variable name in "group_vars/all.yml". After renaming the variable both the old name and the new name get applied during the template run due the inventory not respecting the SCM branch override. The old name is a valid variable that we did not want overridden by "all.yml" for our testing. The end result is that our testing does not represent the actual behavior that ansible should have.

The expected behavior is the when overriding the SCM branch the dependent inventory should also be temporarily updated with the new branch just for that run. This should especially be the case when the inventory source is set to override AWX variables

[AlanCoding]

The expected behavior is the when overriding the SCM branch the dependent inventory should also be temporarily updated with the new branch just for that run.

Under some conditions - the inventory source is set to update on launch. It's an SCM source and shares the same project with the job template. In those narrow conditions this is an agreeable expectation.

[ghost]

The best solution would be for the AWX inventory to not import variables from an SCM source. That way the SCM project branch is correctly respected even when the branch is overridden. This also allows the AWX inventory variables to be treated with a higher precedence so that they can represent values needed to run the project in the AWX environment and not just plain ansible. Also, this would add support for vaulted files in the project which isn't allowed right now.

The work required is limited to reworking the SCM inventory source and wouldn't affect the rest of AWX.