Add Dynamic group intersections to inventory file #10131

Closed
rhartkopf opened this Issue Jan 29, 2015 · 20 comments

Projects

None yet
@rhartkopf
Issue Type:

Feature Idea

Ansible Version:

ansible 1.9

Environment:

N/A

Summary:

I'd like to use group patterns to find the intersection of two dynamic groups, then make those groups the children of other groups. I've tested two methods and either get an error or no results. Method 1 from below is the most intuitive based on the documentation for patterns and dynamic inventory groups.

Steps To Reproduce:

hosts/ec2.py and hosts/ec2.ini: http://docs.ansible.com/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script

hosts/static:

# One static host
[all]
host1

[east]
host2

[west]
host3

# Method 1: Define two static groups composed of one dynamic group
[tag_aws_autoscaling_groupName_Frontend]
[us-east-1]

# Method 2: Define one static group composed of intersection of two dynamic groups
[tag_aws_autoscaling_groupName_Frontend:&us-west-2]

# Control: Set one dynamic group as child
[all:children]
tag_aws_autoscaling_groupName_Frontend

# Set intersection of dynamic groups as children
[east:children]
tag_aws_autoscaling_groupName_Frontend:&us-east-1

[west:children]
tag_aws_autoscaling_groupName_Frontend:&us-west-2

Expected Results:

🔵 Control:

$ ansible -i hosts all --list-hosts
host1
54.xxx.xxx.xxx (hosts from all regions)

🔵 Method 1:

$ ansible -i hosts east --list-hosts
host2
54.xxx.xxx.xxx (hosts from us-east-1)

🔵 Method 2:

$ ansible -i hosts west --list-hosts
host3
54.xxx.xxx.xxx (hosts from us-west-2)
Actual Results:

🔵 Control:

$ ansible -i hosts all --list-hosts
host1
54.xxx.xxx.xxx (hosts from all regions)

🔴 Method 1:

$ ansible -i hosts east --list-hosts
ERROR: child group is not defined: (tag_aws_autoscaling_groupName_Frontend:&us-east-1)

🔴 Method 2:

$ ansible -i hosts west --list-hosts
host3
@bcoca bcoca changed the title from Dynamic group intersections fail in inventory file to Add Dynamic group intersections to inventory file Feb 2, 2015
@sanear
sanear commented Jul 17, 2015

I would absolutely love to have this - I have a lot of ugly-as-sin J2 loops performing this functionality right now (for host in groups[group a] if host.{some_tag} == {tag_value}, etc), and having actual datastructures would be far preferable.

@ckonkel
ckonkel commented Aug 3, 2015

Method 2: Define one static group composed of intersection of two dynamic groups

[tag_aws_autoscaling_groupName_Frontend:&us-west-2]

This would be a great addition! I find almost all my deployments and provisions have intersections... Rather then do this in a customized dynamic hosts file, would love to see in static hosts file!

@archit
archit commented Aug 8, 2015

+1 for this feature.

@jan-g
jan-g commented Aug 9, 2015

My recent questions about this are related to this. My need is to
generalise the notion of run_once: I have a group that is "sliced
horizontally" into a number of dynamic groups. I need to be able to run a
play on one member of each "slice". CConstructs using group[0] are a poor
substitute since I want to target active, up hosts - not every member of
the group is guaranteed to be so.

Suggestions welcome.

On Friday, July 17, 2015, Andrew Sannier notifications@github.com wrote:

I would absolutely love to have this - I have a lot of ugly-as-sin J2
loops performing this functionality right now (for host in groups[group a]
if host.{some_tag} == {tag_value}, etc), and having actual datastructures
would be far preferable.


Reply to this email directly or view it on GitHub
#10131 (comment).

@MadGhigno

I have 400+ servers on EC2 and I make large use of Tags to classify VMs, having support for intersection would improve Ansible flexibility in a such large environment.

+1

@srvg
Member
srvg commented Aug 26, 2015

Interesting idea. Definitely needs a bigger discussion on 'dynamic groups'.
I'd love to find a way to dynamically have a -dev -test and -prod subgroup of each project group for instance (which I define manually now)

@MadGhigno

My problem is that many servers are managed by auto-scaling groups and this makes the manually defined groups obsolete in less than one day.

@srvg
Member
srvg commented Aug 26, 2015

On 26 August 2015 at 20:27, MadGhigno notifications@github.com wrote:

My problem is that many servers are managed by auto-scaling groups and
this makes the manually defined groups obsolete in less than one day.

​I understand this is how Amazon stuff seems to work. To be honest, I never
understood how such a thing can work. I'm used to create hosts from the
inventory, not the other way 'round.

Given, I do consider the ansible inventory to be the start of pretty much
everything. Some day I hope to make it a Single Source of Truth.. But that
is my point of view.

@bcoca
Member
bcoca commented Aug 26, 2015

you can use the current group_by task to do this, for example, simple intersection of group1 and group2:

- hosts: group1
  tasks:
    - group_by: key=group_intesection
      when: inventor_hostname in groups['group2']
@tartansandal

Nice trick Brian.

So one approach would create a 'subgroups.yml' playbook that, when run, defines the dynamic intersection groups depending on the current state of your AWS account. We could then include that at the top of any playbook that uses those groups. We'd probably want to set gather_facts: false so that this runs reasonably fast.

From a modelling perspective, this highlights that your infrastructure is actually dynamic and not static as the hosts file approach would suggest.

I think to get this to work with group_vars, you would have to predefine empty (placeholder) intersection groups in your hosts file like so:

# predefine tag group for test environment
[tag_Env_test]

# predefine tag group for service application
[tag_App_service]

# predefine tag group for intersection of the above groups
[tag_Env_test_App_service]

@sanear
sanear commented Sep 29, 2015

Just want to update on Brian's trick, which I have gotten working for my
own purposes.

You specifically must not add predefined tag groups for the intersections
in your hosts file - if you do, those hardcoded empty groups will
override those created in the subgroups.yml.

My working solution is a subgroups file with entries like so:

hosts: tag1:&tag2[:&othertags...]
gather_facts: no
group_by: tag1_tag2

One entry like that for each desired subgroup, where tag1_tag2 is the
name you want for the group. I couldn't get Brian's example of group_by
with when to work, unfortunately, but this seems to do the trick.

Just include this playbook at the top of a playbook that references your
new group names, and everything should work out just fine.

On Thu, Sep 17, 2015 at 6:25 PM, Kal Hodgson notifications@github.com
wrote:

Nice trick Brian.

So one approach would create a 'subgroups.yml' playbook that, when run,
defines the dynamic intersection groups depending on the current state of
your AWS account. We could then include that at the top of any playbook
that uses those groups. We'd probably want to set gather_facts: false so
that this runs reasonably fast.

From a modelling perspective, this highlights that your infrastructure is
actually dynamic and not static as the hosts file approach would suggest.

I think to get this to work with group_vars, you would have to predefine
empty (placeholder) intersection groups in your hosts file like so:

predefine tag group for test environment

[tag_Env_test]

predefine tag group for service application

[tag_App_service]

predefine tag group for intersection of the above groups

[tag_Env_test_App_service]


Reply to this email directly or view it on GitHub
#10131 (comment).

@tartansandal

On 30 September 2015 at 07:38, Andrew Sannier notifications@github.com
wrote:

You specifically must not add predefined tag groups for the intersections
in your hosts file - if you do, those hardcoded empty groups will
override those created in the subgroups.yml.

My working solution is a subgroups file with entries like so:

hosts: tag1:&tag2[:&othertags...]
gather_facts: no
group_by: tag1_tag2

One entry like that for each desired subgroup, where tag1_tag2 is the
name you want for the group. I couldn't get Brian's example of group_by
with when to work, unfortunately, but this seems to do the trick.

​Thanks for the update Andrew.

I could not get group_by and when to work​ with or without "predefined
subgroups". After a number of iterations. I did get add_host to work
with the same form that you are using, but with "pre-defined subgroups",
but could not get when to work in any of my attempts.

As I think more on it, empty "pre-defined subgroups" makes much less sense
from a modelling perspective: they should only exist when dynamically
created!

I'll give your approach a big +1, being shorter and less confusing.
Perhaps this trick should make its way in to the ec2 FAQ?

K.

@jimi-c jimi-c removed the P4 label Dec 7, 2015
@carmi
carmi commented Dec 30, 2015

+1 I've also run into this issue where I'd like to intersect two dynamic groups.

@toidi
Contributor
toidi commented Jan 7, 2016

+1

@arturmartins

+1!

@cobyrne09

+1

@flamingspaz flamingspaz referenced this issue in mozilla/partinfra-playbooks Feb 15, 2016
Merged

Fix inventory #1

@pporada-gl
pporada-gl commented Jun 5, 2016 edited

Using these tricks feels really awkward from a user experience perspective. Does anyone have a more complete example of @sanear's trick?

@jimi-c
Member
jimi-c commented Jul 12, 2016

Switch to Proposal

Hi!

As of April of 2016, we have started using the Ansible Proposal process for large feature ideas or changes in current functionality, such as this. Because this issue has been open for so long without any action, we are going to go ahead and close it at this time. If you are still interested in seeing this new feature get into Ansible, please submit a proposal for it using this process.

https://github.com/ansible/proposals/blob/master/proposals_process_proposal.md

If you have any further questions, please let us know by stopping by our devel mailing list, or our devel IRC channel:

Thank you!

@jimi-c jimi-c closed this Jul 12, 2016
@mazubieta

Hi Everybody!

This problem has been something that has bugged me since picking up Ansible. I've monitored this thread, and was really disappointed when it was closed the way it was. (by the way, I could not get any of the described work arounds working. I'm pretty sure some of the snippets aren't even valid Ansible).

So, I solved it by realizing that dynamic inventory is just JSON consumed via stdin. So I wrote something that takes the output of another dynamic inventory file, reads in a custom static inventory file that allows usage of Ansible Patterns, and transforms the inventory JSON and outputs the result to stdout.

ansible-meta-dynamic-invetory repository

@pporada-gl @cobyrne09 @toidi @carmi @tartansandal @sanear @srvg @MadGhigno @archit @ckonkel @rhartkopf

@jimi-c
Member
jimi-c commented Sep 19, 2016

@mazubieta I understand your concerns, however the issue had not had any movement on it in almost a year (besides +1's). Moving it to a proposal is not rejecting the idea, it's asking for it to be moved to a format where the community can vote on it and (if it is accepted) to be placed on a schedule for implementation.

@gerrit-ovirt-org gerrit-ovirt-org pushed a commit to oVirt/ovirt-engine that referenced this issue Jan 10, 2017
@didib @sandrobonazzola didib + sandrobonazzola packaging: ovirt-engine-hosts-ansible-inventory: Add up_metrics hosts
When originally writing this, I thought it will be easy to do the
intersection between 'ovirt_metrics_hosts' and 'ovirt_up_hosts' in
ansible itself. Appanrently that's not so easy, doing this here for now.

See also: ansible/ansible#10131

Bug-Url: https://bugzilla.redhat.com/1405813
Change-Id: I3fa8d34f30fb553a298b4ff7d419c4ecfad7d61b
Signed-off-by: Yedidyah Bar David <didi@redhat.com>
4e92536
@gerrit-ovirt-org gerrit-ovirt-org pushed a commit to oVirt/ovirt-engine that referenced this issue Jan 17, 2017
@didib didib packaging: ovirt-engine-hosts-ansible-inventory: Add up_metrics hosts
When originally writing this, I thought it will be easy to do the
intersection between 'ovirt_metrics_hosts' and 'ovirt_up_hosts' in
ansible itself. Appanrently that's not so easy, doing this here for now.

See also: ansible/ansible#10131

Bug-Url: https://bugzilla.redhat.com/1405813
Change-Id: I3fa8d34f30fb553a298b4ff7d419c4ecfad7d61b
Signed-off-by: Yedidyah Bar David <didi@redhat.com>
(cherry picked from commit 4e92536)
7995100
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment