Smart Inventory - Searching on host variables.

[wtcross]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

• API
• UI

SUMMARY

As an AWX user I should be able to create a smart inventory smart host filter based on host vars associated with a host. The behavior of this feature should be the same as how a user can create a smart host filter based on ansible_facts.

STEPS TO REPRODUCE

A dictionary called something like host_vars should be created that can be searched like this with a smart host filter:

host_vars.my_cool_hostvar:foo

ACTUAL RESULTS

I would expect this to only result in showing hosts that have a host variable my_cool_hostvar set to foo.

[wtcross]

The motivation of this feature request it to be able to create smart inventories based on variables created via dynamic inventory scripts. Think integration with Puppet DB or the ServiceNow CMDB.

[AlanCoding]

I don't see what the shortcoming is.

GET /api/v2/hosts/?host_filter=variables__icontains=foobar

When I do this, I see my host that I put the foobar variable into.

https://gist.github.com/AlanCoding/fc54f5557cc1ae19fdbf3582ac7d0de5

[chrismeyersfsu]

This would absolutely be an awesome additional; applying the json host_filter syntax to host vars.

Three things to consider.

1. We added a feature in tower like 2.2 to save comments in host vars. So we store them as strings basically. This is easily gotten around by continuing to store them the way we do currently and a new, shadow, json field.
2. Database indexing size considerations (see below).
3. Implicit host vars that come from groups and inventory.
   (3a.) differentiating from the want to search inclusively through the inv+group+host vars or exclusively, only the host vars.
   (3b.) Being able to support the inclusive search.

---

GIN index on 100,000 records w/ 2-4 KB average JSON size (typical ansible gather_facts payload).

Hosts	Table Size	Index Size
100,000	10 MB	2.483 GB

[wtcross]

@AlanCoding The issue with your example is that it's doing a simple search on a single word. A user will want to search for a specific variable-value combination. A good example is something as simple as the ec2_state host variable created by the ec2 inventory source.

ec2_state: running

You'd have to search for both the key and value. On top of that I'm not sure how good of an idea it is to allow searching for variables using icontains when the value of one variable could contain the target text. A comment could even include the searched text.

[wtcross]

@chrismeyersfsu Those are three very good points. I like the idea of the json column. Whatever code is running the inventory script could simply strip out the comments and then store the json in the corresponding column for that host. This would make it host-specific and remove any overrides from group/inventory variables, but I think that might be ok. I see the primary use case as being used to search on variables returned by the inventory source itself.

Thoughts?

[AlanCoding]

@wtcross You are right. If the variables come from an inventory import, then we can guarantee they will be JSON (again, assuming a user did not modify them), and it appears that the host filter rejects any ", so there would be no way to even match a shallow key: value combination.

[rahst12]

A couple things -

1. In the Smart Inventory, the "Key" button says to use "facts" not "ansible_facts" to access Facts. That should be updated.
2. Variables can already be accessed, but only as the entire json. A primary use case, when using Open Stack, is the following:
   1. Create a Dynamic Inventory of All Hosts, named "All Hosts"
   2. Create a Smart Inventory, based on "All Hosts, filtering on only "Active" hosts, named "All Active Hosts".
      1. "variables.openstack__vm_state:active"

There are many instances in OpenStack where servers might be powered down, suspended, stuck in a rebooting cycle, etc. I need to be able to apply a template to only Active Servers, so I don't see a false-positive failure based upon an inability to SSH to the box.

Link to my StackOverflow Question about this: https://stackoverflow.com/questions/47100910/whats-the-syntax-for-using-facts-in-ansible-tower/47316839#47316839

[mruzicka-redhat]

This appears to be working in the API these days, but not in the UI due to the way that the search is processed. Any chance we might have this available in the UI in the future? We have a large Ansible Tower customer looking for this functionality.

[wenottingham]

The only way it's working in the API is by a free-text search of the variable blob, that will break if the variables are switched between JSON/YAML, and cannot be used to search complex variables (among other things). It's not productizable in its current state.

[rahst12]

This would be great if it were implemented. I'm also using Ansible Tower and this is one of a number of requested features.

[alexander-bauer]

This would be extremely useful for people using the Foreman dynamic inventory. Foreman parameters are exposed in the host variables, and those have a lot of potential for use in smart inventories.

[jasweet]

We've requested this feature as well. Would be nice to see it expedited.

[rahst12]

This appears to be working in the API these days, but not in the UI due to the way that the search is processed. Any chance we might have this available in the UI in the future? We have a large Ansible Tower customer looking for this functionality.

Have you gotten any traction on this. We're also a paying Ansible Tower user looking for this same feature... @mruzicka-redhat

[mruzicka-redhat]

@rahst12 this appears to be on the list, but there is no current ETA. If you have a TAM it would be good to raise this with them too. In either case, it might be a good idea to open a support case to request this addition, if you haven't already. It may help the team prioritize based on customer demand.

[rahst12]

@mruzicka-redhat What is a TAM?

[mruzicka-redhat]

@rahst12 Sorry, out of context and unrelated to this this thread. A TAM is a Technical Account Manager, which is an add on service through Red Hat. Not required, but TAMs (I am one BTW) can help advocate on feature requests (and plenty of other things).

Also an FYI, it seems my earlier comment about it working in the API is not really accurate. This isn't supported and may not return expected results.

[rahst12]

@mruzicka-redhat I have heard of a TAM then... Our sales rep said we need to work with him/her because we've got 7 GitHub issues that would be very very helpful to us if implemented:

Add IP/FQDN to Hosts Tables | #662
UX improvement: Increase visibility of the ad hoc command feature. | #612
Incorrect opacity on SMART INVENTORY and RUN COMMAND buttons in Inventories. | #659
Playbook fails when ssh host key changes | #452
Smart Inventory - Searching on host variables. | #371
Gitlab (self-hosted) SSO | #94
Documentation Updates for Smart Inventory -> Smart Host Filter -> Dynamic Host | #663

I'm trying to get something setup to meet our TAM. A interesting question though - I'm not clear anymore how much of AWX/Tower is contributed to via the general community, and how much is driven by RedHat employees.

[mruzicka-redhat]

Bumping to see if there might be any additional movement on this request. Thank you.

[jagibson]

I'm surprised this doesn't work. It seems like basic functionality for smart inventories IMHO.

[dellarte]

Hello guy, since this feature isn't implemented yet, do you have any idea how can i filter only two hosts (choosed by human) amoung my WMWare dynamic inventory ?

[jasweet]

@dellarte
Your question, as worded, doesn’t seem relevant to this thread. You could use “limit” for what you’ve asked.

[dellarte]

@jasweet
I probably did not express myself well. I have created a Synchronized Vmware Inventory which lists my virtual machines in "hosts" very well.
I added custom variables directly on choosen host through the "details" page of an host in order to be able to filter this host in a smart inventory.
e.g. If I set a "upgradable: true" variable on 2 of my hosts (still available from vmware inventory), i would appreciate i can retrieve only this 2 hosts with a simple filter.
Since this is not implemented yet, i asked if anyone got a tips to do this

[jasweet]

You can build your inventory, using a dynamic script or otherwise, and then create a playbook that caches the desired host variables as ansible facts. If using a dynamic inventory script, and such a playbook, and you don't want to run two separate jobs, you can create a workflow and incorporate each into the workflow. Then you have a single call to kick it off.

…

On Tue, Jun 18, 2019, 6:54 AM gg ***@***.***> wrote: @jasweet <https://github.com/jasweet> I probably did not express myself well. I have created a Synchronized Vmware Inventory which lists my virtual machines in "hosts" very well. I added custom variables directly on choosen host through the "details" page of an host in order to be able to filter this host in a smart inventory. e.g. If I set a "upgradable: true" variable on 2 of my hosts (still available from vmware inventory), i would appreciate i can retrieve only this 2 hosts with a simple filter. Since this is not implemented yet, i asked if anyone got a tips to do this — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#371?email_source=notifications&email_token=AHK2NJW2V76NVBRLSUJJLXTP3DEH3A5CNFSM4D56MZ5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX6EFII#issuecomment-503071393>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHK2NJTLL43ZMEG5YQMD4WDP3DEH3ANCNFSM4D56MZ5A> .

[jasweet]

Abandon all hope. This thread is dead. RH hasn’t taken this seriously for all these years when anyone that really uses Ansible AWX/Tower inventories needs this.

boooooooo!

[DII-dsward]

Not having this is causing my companies new adoption of Ansible Tower to consume multiple licenses as a result of not being able to use variables when creating smart inventory in the UI. This should have been fixed in 2017!

[jasweet]

Not having this is causing my companies new adoption of Ansible Tower to consume multiple licenses as a result of not being able to use variables when creating smart inventory in the UI. This should have been fixed in 2017!

Our team solved this by no longer using Ansible Tower/AWX.

[DII-dsward]

Not having this is causing my companies new adoption of Ansible Tower to consume multiple licenses as a result of not being able to use variables when creating smart inventory in the UI. This should have been fixed in 2017!

Our team solved this by no longer using Ansible Tower/AWX.

just curious, what did your company end up replacing Tower with?

[jasweet]

Not having this is causing my companies new adoption of Ansible Tower to consume multiple licenses as a result of not being able to use variables when creating smart inventory in the UI. This should have been fixed in 2017!

Our team solved this by no longer using Ansible Tower/AWX.

just curious, what did your company end up replacing Tower with?

We didn't replace it. We just stopped using Tower.

Also, there are workarounds to your license consumption challenge, not that we'll discuss that here. You're RH consultant/support folks should be able to help.

[jagibson]

We didn't replace it. We just stopped using Tower.

So frustrating. We never started using tower - not having this functionality was the last straw when I was evaluating.

[HOSTED-POWER]

We would love to see this feature as well ... :)

[daemenseth]

what is the state of this feature

[toweransi]

I'd like to underscore the power of this requested feature in an enterprise with complex inventories.

For any dynamic or static inventory, if a host has a var for "application_name", that var can filter a smart inventory, and boom you have effectively targeted the hosts of that app. You can rapidly target/filter an application, then filter further with "environment", "division", and then filter for a desired ansible fact.
This is extremely fast and effective within the UI.

Now this does work as a smart inventory filter using "variables.icontains:".
However, this comes up short with 2 potential issues cited above, and IMHO is not quite par for an enterprise-grade solution.

(AFAIK) this still remains vulnerable to 1) mention above by [AlanCoding commented on Oct 6, 2017], where JSON-to-YAML conversion might break the filter(?) and 2) mention by [wtcross commented on Oct 6, 2017} where the flat search of the value could show up elsewhere in a comment.

An enterprise-grade implementation would be to search the keypair, not just scraping everything for a value that you hope remains unique. (Although this is exactly what I am doing in Tower in my enterprise now).

I would say that any complex enterprise using AWX or Tower who has not asked for this, is probably a good opportunity for a RedHat TAM to take them deeper into inventory filtering, which could create considerable value.

Thanks for listening.

[gigi1983]

ciao, we were able by ourself and no support to implement smart inventories basing on hostvariables. it works basically using regex on hostvars. the ui seems to do not support this but we implemented the creation of smart inventories using surveys.
For what I got....this is not intended to be given or supported in short time by developers.

[gigi1983]

ciao, we were able by ourself and no support to implement smart inventories basing on hostvariables. it works basically using regex on hostvars. the ui seems to do not support this but we implemented the creation of smart inventories using surveys. For what I got....this is not intended to be given or supported in short time by developers.

as u got we had to develop (my company) by ourself because at least about me facts are useless. but in addition to the smartinventory kind of filter there are a lot at least about me issues related to the rbac model

[daleroux]

Hi team, what's the status on this feature?

[AlanCoding]

I am proposing #13303 as an alternative to solve the problem here. Instead of smart inventory, it would solve the problem by passing multiple inventories to the job run, and using the constructed inventory plugin from core to create groupings based on hostvars.

Looking to get feedback, will bring this up at the next AWX Office Hours.

[jasweet]

I am proposing #13303 as an alternative to solve the problem here. Instead of smart inventory, it would solve the problem by passing multiple inventories to the job run, and using the constructed inventory plugin from core to create groupings based on hostvars.

Looking to get feedback, will bring this up at the next AWX Office Hours.

I think one of the things this needs to address is the desire to filter hostvars on dynamic inventories built via dynamic inventory scripts. I haven't worked in Ansible for a couple years, however, iirc, we were using a script to build an inventory of all network devices in a given environment/topology. For example, a Clos IP fabric. We'd then desire to run specific jobs/tasks against devices based on their particular role in that topology. We found longer ways to do it with workarounds, but filtering on hostvars would have been a clear win for us. I wrote the inventory script, so can try to look back and discuss further if I still have access to that repo.

Hope your suggestion will solve that challenge, and others, that people are still facing years later.

[AlanCoding]

I've put together a demo problem for searching on hostvars and using logic, that is in this folder so that it can be imported as an SCM inventory source:

https://github.com/AlanCoding/Ansible-inventory-file-examples/tree/master/issues/AWX371

I updated the proof-of-concept #13303 so that the new kind of "constructed" inventory is presented in a similar manner to current "smart" inventory.

[shanemcd]

We are moving away from smart inventories and recently introduced a new feature called Constructed Inventory. That merged in #13448 and got released with AWX 22. Constructed Inventories allow for querying host vars. I'm going to go ahead and close this and can also follow up with a link to a blog post that @AlanCoding is currently working on.

[gamuniz]

Docs to new constructed inventory source:

https://github.com/ansible/awx/blob/devel/docs/inventory/constructed_inventory.md

[cidrbl0ck]

Whats the average timeframe for something like this making it's way into AAP? I too would have loved Smart Inventories by tags or key-value pairs.

[AlanCoding]

what do you mean "by tags"?

[cidrbl0ck]

Other than tags being a super common, key:value set used in tons of applications, and in normal inventory sources. I could see defining a Smart Inventory by tag as useful too.

[johnrmarley]

I've found an OK work around. As others have said, the host variables are buried in a single JSON string. You can use regex or iregex to match on this string. Only problem is your regex could match anywhere. You really want to be able to match on a regex like /"variable_name": "value"/ (without the slashes), but the quotes will break it.

So, replace the " with '.'! That is, a regex like /variable_name.: .value/. The . matches anything of course, but the ": " between the variable name and the value helps.

Of course it's not perfect, but if you're careful you should be OK!

[Klaas-]

I've found an OK work around. As others have said, the host variables are buried in a single JSON string. You can use regex or iregex to match on this string. Only problem is your regex could match anywhere. You really want to be able to match on a regex like /"variable_name": "value"/ (without the slashes), but the quotes will break it.

So, replace the " with '.'! That is, a regex like /variable_name.: .value/. The . matches anything of course, but the ": " between the variable name and the value helps.

Of course it's not perfect, but if you're careful you should be OK!

I think you can more easily use constructed inventories. They are new in 22.0.0