Smart inventories don't contain groups

[freehck]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

• API
• UI

SUMMARY

Smart inventory contains now only hosts, but not their groups. This creates problems for roles that use group names.

ENVIRONMENT

• AWX version: 1.0.6.11
• AWX install method: docker on linux
• Ansible version: 2.5.3
• Operating System: Ubuntu 18.04 LTS

STEPS TO REPRODUCE

• Make an inventory Inv with some number of hosts: Ha, Hb and Hc
• Make a group Gtop
• Make some subgroups Gsub1, Gsub2, ... in Gtop
• Make new smart inventory Smrt with host filter by group Gtop
• Make a job template, and set Smrt as its inventory
• Run this job template

EXPECTED RESULTS

I expected to see groups Gtop, Gsub1, Gsub2 and so on: all the groups from Inv the filtered hosts in the Smrt are members of.

ACTUAL RESULTS

Only hosts are in the Smrt.

[wenottingham]

Smart inventories don't have groups by design; this would be a large change to how they're constructed on the backend.

[freehck]

@wenottingham so what are Smart Inventories needed for?
I mean, I can't imagine how anyone uses inventory without groups.

[AlanCoding]

We are making some changes to the inventory script generation for job slicing, and I think this is a good opportunity to implement this request as well. The particular solution I have in mind I have discussed with @wwitzel3, and it comes down to a customized version of this query:

awx/awx/main/models/inventory.py

Line 248 in 5910b8c

group_hosts_kw = dict(group__inventory_id=self.id, host__inventory_id=self.id)

Specifically designed for smart inventories, so we don't filter on inventory, but instead host__in=self.hosts, or something like that.

This assumes that a particular implementation is being requested, and I am somewhat unclear on that. Our patch would be designed to returned host-group memberships from source inventories, it would not be allowing users to group hosts inside of the smart inventory. The issue wording seems to reflect this design, but I don't know that everyone agrees on it.

Also, consider that this patch will have a known limitation. In the case that 2 source inventories had a group of the same name, it would be treated as the same group for the purposes of the smart inventory (very important detail). However, annotating the group-group relationships would be borderline madness. You simply can't delineate which group is which, prevent cycles, etc. For these reasons, I think we should omit group-group (child-parent) relationships in this patch.

@wenottingham does this design & stated limitation sound good to you?

[wenottingham]

So it would be a flat hierarchy?

How performant would this be?

[AlanCoding]

Performance is the main technical issue to be addressed, we already know pretty well what the code change would be. There are some existing large inventory examples that we will need to test with this (and be sure that groups are involved).

[jaevans]

Any progress on this change? This is huge issue for us, causing us to need to maintain a number of different manually created inventories to get around the lack of groups in smart inventories. We'd be willing to take a reasonable performance hit to get the feature working.

[wenottingham]

It's not just performance, there are also questions of semantics.

Having a host/playbook behave differently in a smart inventory because it's now getting a different copy of group vars (as mentioned above) would likely be surprising, and there's no good workaround.

[R-0-Y]

It's not just performance, there are also questions of semantics.

Having a host/playbook behave differently in a smart inventory because it's now getting a different copy of group vars (as mentioned above) would likely be surprising, and there's no good workaround.

Hi, and if instead of the group name, we use the group id ? With groups:N insteand of groups.name:group_name
In this case there wouldn't be any problems regarding group name.

Moreover, in our case, we only want a global inventory from our vmware infrastructure with tagged vm. And smart inventory were supposer to get all inventory needed for us. But without group_vars or group available, like some already said here, it's not really usable... :/

[wenottingham]

Hi, and if instead of the group name, we use the group id ? With groups:N insteand of groups.name:group_name
In this case there wouldn't be any problems regarding group name.

Doesn't help - an inventory is manifested as json, with groups containing hosts. You'll still end up with multiple 'groups' with the same name.

[sdktr]

Would be really nice to have group info concatenated into a smart inventory when merging regular inventories.

[vijeshkk]

Hi, and if instead of the group name, we use the group id ? With groups:N insteand of groups.name:group_name
In this case there wouldn't be any problems regarding group name.

Doesn't help - an inventory is manifested as json, with groups containing hosts. You'll still end up with multiple 'groups' with the same name.

I think this applies to hosts as well. bcoz if the host is present in multiple inventories, it still pulls that host to smart inventory, and still manages to avoid duplicates. Shouldn't we be able to apply similar logic for groups?

[vijeshkk]

I think, without having possibility to group hosts and use group_vars, smart inventory doesnt makes sense to most of our production scenarios.

[vjsaisha]

This feature is important one. Give a priority. This will help in rolling out the new version code in production environment in controlled phase.

[Platou]

Same feature request here

[friendlypenguin]

This is an important feature for us as well.

[cwaninger]

Need to build ugly workarounds due to this.
Playbook runs against a predefined group. Smart Inventory picks out all hosts depending on facts/variables. Sadly i can not run the playbook due to missing groups.
Would really appreciate if this would be implemented somehow

[HOSTED-POWER]

I was testing new dynamic inventories and was about to combine them in a smart inventory, however without groups they are useless to use for the moment :(

[gearboxscott]

Has there been any agreement or movement on this issue?

[jessechoward]

I will echo others. Without at least group membership, smart inventories are not useful.

Fundamental to the use of smart inventories is the idea that a single dynamic inventory can be broken down into smaller smart inventories for smaller subsets of work. I will admit my frustration comes from the fact it is IMPLIED that group membership and group_vars are included in smart inventories. https://docs.ansible.com/ansible-tower/latest/html/userguide/inventories.html#smart-inventories. If it was updated to EXPLICITLY tell users it DOES NOT include group membership or group_vars, it wouldn't fix the issue, but may lessen my frustration in thinking I did something wrong or there is a way to enable it.

[it-pappa]

I just found out it is not possible to have more then one group in regular inventorys and using source.

awx 17.0.1.
When trying to use inventory - Sourced from project i just found out it is not possible to add an inventory file with more then just one group.

[Q-Hilbtec]

Staggering that this is still an issue - this really seems like a core feature request and has been requested since 2018.

[ebarrere]

I am honestly blown away that this would have been by design. I agree with the comments above — how is this "feature" helpful if it doesn't support a key piece of ansible inventories?

[it-pappa]

I am honestly blown away that this would have been by design. I agree with the comments above — how is this "feature" helpful if it doesn't support a key piece of ansible inventories?

I have wrote an API script to create regular inventories etc. But it is not the best way since i cant drop the inventory afterwords.

[ffirg]

Watch this space, something is coming :)
https://www.ansible.com/blog/ansible-automation-platform-moving-towards-smarter-inventory

[shanemcd]

We're working on this here: #13448

[ebarrere]

Thanks for the feedback; I’m looking forward to this feature. In the meantime, with a modified EE, would there be anything preventing us from using constructed inventories “under the covers”?

[AlanCoding]

^ that would be challenging, because we run each inventory source through ansible-inventory individually, and constructed needs to be passed as a -i option as the last of several.

[ebarrere]

Got it. okay, I’ll wait for the release.

[fivetran-joliveira]

As group vars support on smart inventories has not yet been implemented, I was wondering if we could achieve a similar result by having multiple dynamic inventories (the non-smart ones 😄 - the ones that actually work with group vars) and each one having a single inventory source applying a host filter/group filter.

However I couldn't find anywhere an example of the syntax to be used on inventory source host filter.
I tried ansible host pattern/limit syntax, tried the smart inventory host filter syntax, ..., but none of them works for me.

• Could anyone please share an example of working inventory source filter so we can use regular inventories + filtered inventory sources as an alternative to the yet to be implemented smart filters with group vars?

Example, that does not work:

(groups__name=group_1) and (groups__name__icontains=another_group)

[pidreher]

@fivetran-joliveira reading that tooltip makes me believe that this does exactly what it says. "matching host names" to your provided regex. For example: webserver.*

Was digging around in the source code a bit. And I have absolutely no clue if that is actually where this gets applied, but it at least looks like that for me:

awx/awx/main/utils/mem_inventory.py

Line 143 in 1411d11

if self.host_filter_re and not self.host_filter_re.match(host_name):

From what you've described you are planning to do, I think you could just use the 'groups' parameter, which all inventory plugins should share.
I recently added an example to the vmware docs:
https://github.com/ansible-collections/community.vmware/pull/1643/files

hope that clears some things up. (.. and doesn't spread misinformation.... ?)

[fivetran-joliveira]

Hey @BigPii thanks for your message!
I actually already have groups defined in my inventories and all hosts are grouped.
What I couldn't figure out is how to filter groups using the regular inventory (not smart inventory) Host filter.

So let's say I have my hosted grouped into group_1, group_2, groupA, and group_ABC, what's the syntax to be used in the inventory host filter to exclude group_2 and keep hosts from group1, groupA and groupABC?

Here is the syntax that I tried, and didn't work for regular inventories (although this syntax works for smart inventories):

(groups__name=group_1) and (groups__name__icontains=groupA)

Could anyone point me out to any documentation describing the valid syntax to be used in regular inventory host filter?

[fivetran-joliveira]

I was hoping to understand the valid patterns accepted on inventory host filters by looking at their unit tests, but it seems there is no code coverage for host filter related mehods: https://github.com/ansible/awx/blob/devel/awx/main/tests/unit/utils/test_mem_inventory.py

[pidreher]

Hi @fivetran-joliveira
The host filter simply filters >hostnames< with the provided regex. No groups involved whatsoever.

[ffirg]

I'm going to close out this issue as we have no intention of doing this in smart inventories. In fact it can't be done and that's the reason we're developing the new constructed inventory feature (see comments above).

[freehck]

I see it's closed, but it's unclear without any resolution tag: is done/wontfix/moved?

Yes, I've read you're working on something you call "constructed inventories". But I can't find anything about it in the docs.

• are "constructed inventories" released?
• where's the epic for this feature?
• what issue shall people track to find out the state of the feature?

@ffirg

[ffirg]

@freehck it went into https://github.com/ansible/awx/releases/tag/22.0.0