Add Dynamic group intersections to inventory file

[rhartkopf]

Issue Type:

Feature Idea

Ansible Version:

ansible 1.9

Environment:

N/A

Summary:

I'd like to use group patterns to find the intersection of two dynamic groups, then make those groups the children of other groups. I've tested two methods and either get an error or no results. Method 1 from below is the most intuitive based on the documentation for patterns and dynamic inventory groups.

Steps To Reproduce:

hosts/ec2.py and hosts/ec2.ini: http://docs.ansible.com/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script

hosts/static:

# One static host
[all]
host1

[east]
host2

[west]
host3

# Method 1: Define two static groups composed of one dynamic group
[tag_aws_autoscaling_groupName_Frontend]
[us-east-1]

# Method 2: Define one static group composed of intersection of two dynamic groups
[tag_aws_autoscaling_groupName_Frontend:&us-west-2]

# Control: Set one dynamic group as child
[all:children]
tag_aws_autoscaling_groupName_Frontend

# Set intersection of dynamic groups as children
[east:children]
tag_aws_autoscaling_groupName_Frontend:&us-east-1

[west:children]
tag_aws_autoscaling_groupName_Frontend:&us-west-2

Expected Results:

🔵 Control:

$ ansible -i hosts all --list-hosts
host1
54.xxx.xxx.xxx (hosts from all regions)

🔵 Method 1:

$ ansible -i hosts east --list-hosts
host2
54.xxx.xxx.xxx (hosts from us-east-1)

🔵 Method 2:

$ ansible -i hosts west --list-hosts
host3
54.xxx.xxx.xxx (hosts from us-west-2)

Actual Results:

🔵 Control:

$ ansible -i hosts all --list-hosts
host1
54.xxx.xxx.xxx (hosts from all regions)

🔴 Method 1:

$ ansible -i hosts east --list-hosts
ERROR: child group is not defined: (tag_aws_autoscaling_groupName_Frontend:&us-east-1)

🔴 Method 2:

$ ansible -i hosts west --list-hosts
host3

[sanear]

I would absolutely love to have this - I have a lot of ugly-as-sin J2 loops performing this functionality right now (for host in groups[group a] if host.{some_tag} == {tag_value}, etc), and having actual datastructures would be far preferable.

[ckonkel]

Method 2: Define one static group composed of intersection of two dynamic groups

[tag_aws_autoscaling_groupName_Frontend:&us-west-2]

This would be a great addition! I find almost all my deployments and provisions have intersections... Rather then do this in a customized dynamic hosts file, would love to see in static hosts file!

[archit]

+1 for this feature.

[jan-g]

My recent questions about this are related to this. My need is to
generalise the notion of run_once: I have a group that is "sliced
horizontally" into a number of dynamic groups. I need to be able to run a
play on one member of each "slice". CConstructs using group[0] are a poor
substitute since I want to target active, up hosts - not every member of
the group is guaranteed to be so.

Suggestions welcome.

On Friday, July 17, 2015, Andrew Sannier notifications@github.com wrote:

I would absolutely love to have this - I have a lot of ugly-as-sin J2
loops performing this functionality right now (for host in groups[group a]
if host.{some_tag} == {tag_value}, etc), and having actual datastructures
would be far preferable.

—
Reply to this email directly or view it on GitHub
#10131 (comment).

[MadGhigno]

I have 400+ servers on EC2 and I make large use of Tags to classify VMs, having support for intersection would improve Ansible flexibility in a such large environment.

+1

[srgvg]

Interesting idea. Definitely needs a bigger discussion on 'dynamic groups'.
I'd love to find a way to dynamically have a -dev -test and -prod subgroup of each project group for instance (which I define manually now)

[MadGhigno]

My problem is that many servers are managed by auto-scaling groups and this makes the manually defined groups obsolete in less than one day.

[srgvg]

On 26 August 2015 at 20:27, MadGhigno notifications@github.com wrote:

My problem is that many servers are managed by auto-scaling groups and
this makes the manually defined groups obsolete in less than one day.

​I understand this is how Amazon stuff seems to work. To be honest, I never
understood how such a thing can work. I'm used to create hosts from the
inventory, not the other way 'round.

Given, I do consider the ansible inventory to be the start of pretty much
everything. Some day I hope to make it a Single Source of Truth.. But that
is my point of view.

[bcoca]

you can use the current group_by task to do this, for example, simple intersection of group1 and group2:

- hosts: group1
  tasks:
    - group_by: key=group_intesection
      when: inventor_hostname in groups['group2']

[tartansandal]

Nice trick Brian.

So one approach would create a 'subgroups.yml' playbook that, when run, defines the dynamic intersection groups depending on the current state of your AWS account. We could then include that at the top of any playbook that uses those groups. We'd probably want to set gather_facts: false so that this runs reasonably fast.

From a modelling perspective, this highlights that your infrastructure is actually dynamic and not static as the hosts file approach would suggest.

I think to get this to work with group_vars, you would have to predefine empty (placeholder) intersection groups in your hosts file like so:

# predefine tag group for test environment
[tag_Env_test]

# predefine tag group for service application
[tag_App_service]

# predefine tag group for intersection of the above groups
[tag_Env_test_App_service]

[sanear]

Just want to update on Brian's trick, which I have gotten working for my
own purposes.

You specifically must not add predefined tag groups for the intersections
in your hosts file - if you do, those hardcoded empty groups will
override those created in the subgroups.yml.

My working solution is a subgroups file with entries like so:

hosts: tag1:&tag2[:&othertags...]
gather_facts: no
group_by: tag1_tag2

One entry like that for each desired subgroup, where tag1_tag2 is the
name you want for the group. I couldn't get Brian's example of group_by
with when to work, unfortunately, but this seems to do the trick.

Just include this playbook at the top of a playbook that references your
new group names, and everything should work out just fine.

On Thu, Sep 17, 2015 at 6:25 PM, Kal Hodgson notifications@github.com
wrote:

Nice trick Brian.

So one approach would create a 'subgroups.yml' playbook that, when run,
defines the dynamic intersection groups depending on the current state of
your AWS account. We could then include that at the top of any playbook
that uses those groups. We'd probably want to set gather_facts: false so
that this runs reasonably fast.

From a modelling perspective, this highlights that your infrastructure is
actually dynamic and not static as the hosts file approach would suggest.

I think to get this to work with group_vars, you would have to predefine
empty (placeholder) intersection groups in your hosts file like so:

predefine tag group for test environment

[tag_Env_test]

predefine tag group for service application

[tag_App_service]

predefine tag group for intersection of the above groups

[tag_Env_test_App_service]

—
Reply to this email directly or view it on GitHub
#10131 (comment).

[tartansandal]

On 30 September 2015 at 07:38, Andrew Sannier notifications@github.com
wrote:

You specifically must not add predefined tag groups for the intersections
in your hosts file - if you do, those hardcoded empty groups will
override those created in the subgroups.yml.

My working solution is a subgroups file with entries like so:

hosts: tag1:&tag2[:&othertags...]
gather_facts: no
group_by: tag1_tag2

One entry like that for each desired subgroup, where tag1_tag2 is the
name you want for the group. I couldn't get Brian's example of group_by
with when to work, unfortunately, but this seems to do the trick.

​Thanks for the update Andrew.

I could not get group_by and when to work​ with or without "predefined
subgroups". After a number of iterations. I did get add_host to work
with the same form that you are using, but with "pre-defined subgroups",
but could not get when to work in any of my attempts.

As I think more on it, empty "pre-defined subgroups" makes much less sense
from a modelling perspective: they should only exist when dynamically
created!

I'll give your approach a big +1, being shorter and less confusing.
Perhaps this trick should make its way in to the ec2 FAQ?

K.

[carmi]

+1 I've also run into this issue where I'd like to intersect two dynamic groups.

[ediskandarov]

+1

[arturmartins]

+1!

[colin-byrne-1]

+1

[pporada-gl]

Using these tricks feels really awkward from a user experience perspective. Does anyone have a more complete example of @sanear's trick?

[jimi-c]

Switch to Proposal

Hi!

As of April of 2016, we have started using the Ansible Proposal process for large feature ideas or changes in current functionality, such as this. Because this issue has been open for so long without any action, we are going to go ahead and close it at this time. If you are still interested in seeing this new feature get into Ansible, please submit a proposal for it using this process.

https://github.com/ansible/proposals/blob/master/proposals_process_proposal.md

If you have any further questions, please let us know by stopping by our devel mailing list, or our devel IRC channel:

• #ansible-devel on Freenode
• https://groups.google.com/forum/#!forum/ansible-devel

Thank you!

[iautom8things]

This problem has been something that has bugged me since picking up Ansible. I've monitored this thread, and was really disappointed when it was closed the way it was. (by the way, I could not get any of the described work arounds working. I'm pretty sure some of the snippets aren't even valid Ansible).

So, I solved it by realizing that dynamic inventory is just JSON consumed via stdin. So I wrote something that takes the output of another dynamic inventory file, reads in a custom static inventory file that allows usage of Ansible Patterns, and transforms the inventory JSON and outputs the result to stdout.

ansible-meta-dynamic-invetory repository

@pporada-gl @cobyrne09 @toidi @carmi @tartansandal @sanear @srvg @MadGhigno @archit @ckonkel @rhartkopf

[jimi-c]

@mazubieta I understand your concerns, however the issue had not had any movement on it in almost a year (besides +1's). Moving it to a proposal is not rejecting the idea, it's asking for it to be moved to a format where the community can vote on it and (if it is accepted) to be placed on a schedule for implementation.

[DenverJ]

For anyone else that stumbles upon this issue from a google search before Ansible's official documentation, check out https://docs.ansible.com/ansible/devel/plugins/inventory/constructed.html