-
Notifications
You must be signed in to change notification settings - Fork 23.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
separate group_vars being overwritten when deploying to same host #9065
Comments
I'm also having this issue, it seems like the vars get parsed and attached to the host rather than the staying with the group. Steps To Reproduce: Hosts file:
Run a command on each group:
Expected results: The variable Actual results: host2 outputs the variable from group |
Have the same problem. Example: ./group_vars/group1: ---
name: group1 ./group_vars/group2 ---
name: group2-default hosts file: [group1]
host1
[group2]
host1 So, when I'm trying to run playbook for group1 only, I'm getting this result: Is there any workaround for this? |
inventory vars get merged even if you are not using that group, last group Brian Coca |
@bcoca Why does it work that way? Is this expected behaviour, or an actual bug? |
You could have vars/service(1,2,3,...) files And pass service=1 as extra vars Then for the hosts your targeting in the playbook use the service variable, We do a similar thing for importing vars based on an environment extra var
|
I have same issue on 1.8.4 today. |
Hmm. Seems like the tags 'bug_report' are wrong here. This is not a bug but.. a feature :-) As @hashnz said, "the vars get parsed and attached to the host rather than the staying with the group" which is totally correct. Variables in different groups where the same hosts are member, only makes sense when those groups are in a parent-child relation, the child winning, and it's vars applied. If groups are at the same level, there is not really a deterministic way which one will win. At playbook time targetting a group actually only determines which hosts are in the run, it doesn;t change anything at variable level. The ansible way here would be to set database host and port in a list, and the iterate over it in tasks. |
@srvg Thanks a lot! I got it. I'll try without using group_vars. |
closing as this is not a bug but by design, group vars get flattened per host and do not vary because of how you select the host |
Anyone got a workaround for this? we can't use static vars in the playbook, as we need to differenciate between different environments (handles by different inventory files) and need to run with different variables, depending on the group run, even tough they are run on the same host as another group |
Use a CNAME for the same hostname or an IP address. This is the only
|
It has been fixed but rejected. #6666 The annoying thing is that the ansible guide does recommend to group them by roles http://docs.ansible.com/playbooks_best_practices.html
But if the same host has multiple roles with the same variables you can't use the group variables since you don't know which one will be used. |
I still don't understand how to bind a variable to a specific host for a specific playbook. I want the ability to use the same template, but with different configuration values depending on the environment I'm deploying to. We're already following http://docs.ansible.com/ansible/playbooks_best_practices.html#staging-vs-production, by making different inventories for our different environments Why does this seem so difficult with Ansible? Is Ansible just bad at doing environment based deployments? Please provide a full example, because "The ansible way here would be to set database host and port in a list, and the iterate over it in tasks." does not make any sense to me. Where is the environment specified here? For example, I have an authentication service that has a variable called
Now it's difficult for me to feed that to a common template file. I need to add if else checks throughout my j2 file (and we went down that path, but it was too much duplication and we got rid of it.) I have not found any documentation that handles this case, and it seems like a thing that should not be difficult to do. |
Here's how we do it: AWS instance tagged as:someapi: prod playbook someapi.yml
Execution with extra varsansible-playbook -vv someapi.yml -e "env=prod" roles/someapi/tasks/main.yml
roles/someapi/vars/someapi_prod.ymlauthentication_db_host: 192.168.2.4 roles/someapi/vars/someapi_stag.ymlauthentication_db_host: 127.0.0.1 roles/someapi/templates/config.j2authentication_db_host = {{ authentication_db_host }} HTH, Iain Wright This email message is confidential, intended only for the recipient(s) On Fri, Oct 16, 2015 at 12:42 PM, arianitu notifications@github.com wrote:
|
Can Ansible detect such cases and make a warning with a possible solution hint? |
If I understand correctly how it works -- for future readers like myself who struggle to understand:
|
The main problem is that ansible merge variables even from groups I don't вс, 17 Янв 2016, 20:00, Victor Varvaryuk notifications@github.com:
|
This seems very counter-intuitive. I have an inventory which just has my Vagrant box in each group for testing purposes. But even when I run a playbook which just references a single group, Ansible is pulling in variables from the other groups too which is breaking my testing. |
On 28 April 2016 at 17:55, Philip Wigg notifications@github.com wrote:
Do not look at the inventory as tightly coupled with playbooks. Inventory
HTH, Serge |
I'm bumping into this too. In my case I have a role which installs vhosts. I set the details for the multiple vhosts as an array in my inventory for the server groups, in other words each group can have multiple vhosts. This works well, I define the vhosts for each group and everything seems neat and tidy. The problem comes up when the host is in more than one group, in that case the vhost array in the inventory is overwritten and the last group wins, so only half my vhosts end up being provisioned. An alternative would be to put the vhosts in the hosts instead of in the group, but that doesn't seem clean because I'll have to duplicate for all servers which are in the same group. Can anybody point me in the direction of a better way to do this or to work around it? I feel like I may be abusing the concept of roles or inventory by having my "vhosts" role accept an array of vhosts (a vhost is not exactly a "role" in the sense that you don't say to a server "you are a vhost"). Can anyone offer a better way to do it or share their thoughts? |
By the way, @srvg, thank you for taking the time to respond to me - it was much appreciated and has clarified my understanding. |
My solution is. [service1] [service2] |
This is so stupid. I just spent many hours trying to debug a variable issue, only to discover that it was because of some stupid design philosophy (this). At there very least, there should be a warning when multiple group_var files are loaded for the same host. |
This i very anoying to have group_vars loaded randomly. It make impossible to have inheritance in a such case all > datacenter > environment (integ, devel, prod) > application > host It's very easy to do with puppet hiera but with ansible it's a pain, and, when you have > 100 playbooks/roles to maintain, load variables explicitely is not a must have because you should maintain a vars_file + a group_vars file for the same function.
then gitlab load group vars per name, here the datacenters, after the env, and last the app. |
We filter out closed issues and don't see comments on them, but someone pinged me to respond to some of these questions. If you want attention from devs the mailing list or IRC are better conduits than closed tickets. Ansible looks at groups as properties of a host. For example, you can have a host be part of webserver group (to install a webserver reqs) and part of the northeast_datacenter group (so you know the network gateway, ntp and dns servers to set) also part of the dev_group to install dev tools and point to the non-production database, etc, etc. You CAN use the groups as a way to target hosts but that does not mean the other 'host properties' disappear. I am in the 'males' group and in the 'programmers' group, just because you 'select me' as a programmer I do not stop being male. In the end this is a design decision (not a bug) on how to tackle groups, a way to classify and add properties to hosts. I know other tools use them differently but that is not a reason to change how they work in Ansible, as there is no uniform way ALL other tools use groups, many others look at them this way many don't. This has the limitation of not being able to reuse variable names and only have the 'current group apply' but it has the advantage of always seeing the bigger picture of how the host is defined and makes conflicts easy to spot or resolve. In most of the cases above a use of vars_files or include_vars seems more appropriate than setting up as groups, when talking about application specific variables. Doing include_vars on group_vars ... well that is not how we designed it, but if it works for you ... the caveat is that you are 'double loading' as group_vars are automatically pulled in, this can lead to performance issues on large inventories. We recommend more specific, I hope this clarifies why groups work the way they do in Ansible and why that is different than 'another specific tool', let me know if there is anything I can add to our docs http://docs.ansible.com/ansible/intro_inventory.html to clarify and avoid frustration. I would love to accommodate everyone's way of working, Ansible is very flexible in some areas, but not in this respect and I don't see a good way to make it so nor a sane way to share playbooks afterwards. |
@bcoca is it possible to detect cases when the group vars are being used in a strange way (sorry, cannot formulate this better -- haven't used this for a long time) and make warnings to help users save their time? |
@bcoca What's unclear to me, and I think most of the rest of the commentators on this issue, is what value the Ansible team is looking to preserve by keeping group_var precedence/resolution the way that it is. Can you give an example of a case which is satisfied well by the current solution but would cease to be so if, say, group_vars are only loaded/considered for the groups selected for a given play, rather than for the entire inventory? It seems quite easy to come up with examples which are broken by the way that it is now. |
I understand @bcoca's point as we use it that way too. My biggest frustration with Ansible is it works mostly good for either configuration management or ad-hoc workflows. It doesn't do a great job of either but it's getting better and it has plugin support which I think I will be using to make another issue I have with it better. Most days I still miss Puppet solely for configuration management... Probably the best use case for group_vars is multiple datacenters. Let's say your inventory file looks thusly:
You could use group_vars to set datacenter specific variables that would be inherited by all hosts in that datacenter.
dc1.yml
dc2.yml
Using group_vars like this is fine until you need to merge or override a variable in another group_var file. This is another case where group_vars breaks down. There's no way to set the order in which variables are read in group_vars. A hierarchical layout (like Hiera) that can be determined by facts and not by groups for things like environmental variables comes to mind here but I digress. From my perspective, this use case is mostly good for straight configuration management but it's not so great at deployments or ad-hoc scripts. This is why I would love to see a switch to disable this feature for certain playbooks where you don't want to do this. |
@hanej I'm not sure how that example breaks by the design change I suggested in my previous comment? Given the inventory you've listed, if I I agree that deployments are an issue. Multi-tenanted deployments, specifically. I have two concerns with the way things work now: First, lack of configuration namespacing. Specifically, if I'm maintaining configuration for some applications in a multi-tenanted environment, how do I ensure that someone maintaining a different app which happens to run on the same machines isn't going to add a variable name which conflicts with mine, or that I won't add one which conflicts with theirs? Second, the case everyone keeps talking about here. I'd expect that with the below inventory, I'd never wind up with files written to These two issues are bad enough on their own, but in a large company where many people in different timezones might be editing the same inventory, the current solution seems more than a little risky. Example inventory.
app1_staging.yml:
app1_prod.yml:
|
Don't know if the edit worked to adjust the notification, by my last comment was meant to be directed to @hanej. |
@benjamincburns You asked for an example of using group_vars that satisfies the current loading method but breaks if only using the immediate groups. I provided that with the datacenter example. If Ansible doesn't load variables for all groups, then you cannot set variables, for example, at a datacenter level. I believe this should be a toggle in ansible.cfg or on the command line so you can specify how to load group_vars so it doesn't break the existing use cases. We have multi-tenant environments here and used to get bit by this until I stopped using group_vars. I'm not sure if you've heard of the roles and profiles pattern but it's a really good read and I highly recommend this approach if you're using any kind of configuration management tool. It was written for Puppet but it applies for all tools. You probably wouldn't use this for deployments or ad-hoc stuff but for straight configuration management it's great. |
I use docker hosts so it is quite normal to have different instances of containers in different environments spread across hosts and this bit me. Thanks to @hanej for the workaround and I would also add group vars do not work as advertised and seem to be broken by design. My workaround is to define a play as follows:
Put a vars directory in the playbook dir with group1.yml, group2.yml in there. |
It would be awful if your configurations changed simply because they were accessed differently. Just think of the affect that would have on attempts at convergent architecture. +1 to the Ansible team for keeping variables as consistent as possible. The more interesting question is how you can apply the same role multiple times, sharing some configurations across hosts while extending (and not replacing) others. I think role dependencies allow that, namely by:
So in essence, your wrapper role would include a
When you apply the wrapper role, only the "extendable" configurations are actually changed. The burden is on developers to design their roles in a way that supports this pattern. Ansible team: was this your intention for how roles would be used? I honestly haven't seen a better use-case for dependencies; everything else seems to lead to code soup. |
I just joined victim list today after 1 hour debugging and fortunate enough to found this post. |
+1 This issue affects our software team. |
Unfortunately we also ran into this issue today. Same story, Though we have a generic "tomcat" rol we vary the variables like port numbers based upon the group that is used. However this approach doesn't seem to work. I have no idea yet how to design this. I cant create wrapper roles that have the variables because some of the variables are actually specific to the inventory. Think for example a database password oid. |
I recently found this which eased my pain with this issue: It allows merging of variables no matter where they were defined, so you can merge two dicts/lists in two different groups for example, as opposed to one group winning and overwriting the variable defined in the other groups as is the current behavior. The plugin only works with Ansible 2+. A good article explaining the concept and the use case can be found here. |
We recently come into this issue as well. We have multiple applications running on a single host as different users. For instance: [group_a]
host1
[group_b]
host1 We have no way to specify Our current workaround is to define multiple plays with different - hosts: group_a
ansible_user: user_a
roles:
- my-role
- hosts: group_b
ansible_user: user_b
roles:
- my-role |
@hyojinbae suggested a smart way above to workaround this: [group_a]
user_a@host1 ansible_user=user_a ansible_host=host1
[group_b]
user_b@host1 ansible_user=user_b ansible_host=host1 So now each host has a unique name in the inventory. Not a big fan to put host vars in inventory, but this is the best workaround I have seen so far. |
@hyojinbae I think you are missing the points of hostvars and directives: - hosts: group_a
remote_user: user_a
roles:
- my-role
- hosts: group_b
remote_user: user_b
roles:
- my-role should work as long as you dont set ansible_user. The groups are a property of the host, not an entity to themselves. |
There was a proposal to make it possible to write proper plugins for inventory management. I suspect now that this proposal was closed/finished, it should become possible to implement a dynamic inventory plugin which works around this point of confusion. |
Going to +1 to question the ansible design here. Just as a thought experiment:
Making this small change would make hundreds of DevOps-y people happy, and would hurt nobody. |
From my previous comment, see ansible/proposals#41 for a potential avenue to allow more pluggable behaviour here. |
Can't believe that Ansible do not fix(give hints, warning) or reply more on this issue. |
I encountered this as well. One approach was to use nested variables. So take inventory group my_app_servers for example, and it lines up with the file my_app_servers.yml inside your group_vars, and you've got more than one application to manage on a common host. I nested them like this
In your main.yml, define a loop, and iterate over the nested variable list like this, with_items: "{{ application_env }}" Then in your tasks, you can grab the nested variable that you want, to execute something. |
This issue (#9065) was closed back in 2015. Same #6538. Same #6666. And many more. If everyone will continue using hacks proposed, it will make ansible code ugly:
IMHO these hacks make it less attractive not fully aligned ansible's "Simple IT Automation" motto ;) @bcoca -> Kindly guide what should be the trigger (from community) to have someone reevaluate PR #6666 OR #17236 or something similar that "just works intuitively"? |
I think group_vars themselves should be able to be prioritized. Hiera for Puppet got it right. You can specify how lookups are done based on facts or groups. Let the end user be able to customize how group_vars should be processed and provide a mechanism to easily see the order in which Ansible is applying those variables. |
FYI, we dont normally see notifications on closed issues, responding here because someone pointed me at this, I'm also going to lock so people don't keep commenting in a forum maintainers won't see. @hanej i think you want @ReSearchITEng open a proposal (http://github.com/ansible/proposals), but this is about a fundamental focus of each diff app. Ansible focuses on a task for a host, the other tools you mention focus on a much more complex structure. For ansible 'groups' are just a property of a host, not a principal entity, the other tools focus on groups as 'the main context' and everything else is added onto that. I don't see either approach as 'wrong', just different. Trying to make Ansible work in such a way will break the simplicity paradigm, focus on A host and A task for what is just an organizational focus and transposing what i see as the 'accustomed way of thinking' people bring from other tools. I would argue that this is just a perception on data organization and habits, but that is just my opinion. Feel free to submit a proposal, if you convince and gain enough traction with maintainers you might get the change you want, I just don't think it is a change we need. |
Issue Type:
Bug Report
Ansible Version:
ansible 1.6.1
Environment:
Mac OSX
Summary:
If we have a hosts file:
and we have these group_vars:
and we are running this playbook:
the variables from group_vars/service1 and group_vars/service2 overwrite each other if we're deploying to the same server. This means service1 will get service2 groups variables and get the incorrect database host and port.
to get around these, we've added DNS entries (aliases to www.host1-service.com) so our host looks like:
but this is highly error prone and is not ideal. What are some different methods to getting around this issue? (or misunderstanding of group_vars)
The way I'm doing multi environmental deployments is like this:
Steps To Reproduce:
Summary describes how to reproduce this.
Expected Results:
group_vars should not be overwritten by another group when sharing the same host
Actual Results:
group_vars are overwritten by another group when sharing the same host
The text was updated successfully, but these errors were encountered: