Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to define nodegroups through `master_tops` system #5787

Open
madduck opened this issue Jun 27, 2013 · 27 comments
Open

Ability to define nodegroups through `master_tops` system #5787

madduck opened this issue Jun 27, 2013 · 27 comments
Labels
Milestone

Comments

@madduck
Copy link
Contributor

@madduck madduck commented Jun 27, 2013

Edit: master_tops is probably not the right place for this. I'll repurpose (and retitle) this issue as soon as the storm has left my brain (i.e. "brainstorming is done…").


At least with reclass, nodegroups are part of the inventory and thus could easily be made available to Salt, such that I could target commands or states at groups in my inventory.

Instead of defining a new interface (we already have master_tops and ext_pillar separately, do we want a third?), maybe the master_tops interface could be extended to allow the external source to return nodegroups?

I don't have a clear idea of how this would work yet, but if you give a general green light into this direction, I could start looking and evaluating various options. It is possible that the interface would have to be incompatibly changed, unfortunately, but I would document this appropriately and change the existing tops plugins.

@thatch45
Copy link
Member

@thatch45 thatch45 commented Jun 27, 2013

I think this is a good idea, I am not putting it on the top of my list, but please let me know if you can see a clean way to make this a possibility.
I think that adding it into master_tops makes a lot of sense, since the pillar data maps to a very different context, but ideas will need to be cultivated here to determine exactly how to approach this feature.

@madduck
Copy link
Contributor Author

@madduck madduck commented Jun 27, 2013

Thanks for your time to reply, @thatch45!

@UtahDave and I are just discussing this on IRC and we think that master_tops is not the right place for this, because it just returns the same data as top.sls, and as long as top.sls is not used for node group definitions, master_tops shouldn't be either.

Right now, we are brainstorming about a third external information source, maybe something like an "inventory". It would not only provide node group membership information, but also be able to give a list of all minions and allow the master to report if a minion didn't respond.

We'll keep you posted!

@madduck
Copy link
Contributor Author

@madduck madduck commented Jun 27, 2013

If there were an inventory, then maybe minions that are not addressed by a published command could still respond with a NOOP, or a "yeah, I heard you, but I am not addressed". This would close the loop and allow the master to stay on top of things.

@thatch45
Copy link
Member

@thatch45 thatch45 commented Jun 27, 2013

I am liking where this is going, it will be useful on many levels!

@arthurlogilab
Copy link
Contributor

@arthurlogilab arthurlogilab commented Sep 12, 2013

+1 on this feature

@mgwilliams
Copy link
Contributor

@mgwilliams mgwilliams commented Oct 4, 2013

Inventory could be defined in yaml and/or by an ext_inventory module... I like this idea too. Could this somehow be combined with roster?

@baby-gnu
Copy link

@baby-gnu baby-gnu commented Dec 31, 2013

+1

I can't imagine a configuration management without an inventory of “what I want” to compare to “what I have” and see the progress and configuration policy violations.

@bbinet
Copy link
Contributor

@bbinet bbinet commented Jan 16, 2014

+1

2 similar comments
@nikicat
Copy link
Contributor

@nikicat nikicat commented Jan 24, 2014

+1

@jnials
Copy link

@jnials jnials commented Feb 7, 2014

+1

@loz-hurst
Copy link
Contributor

@loz-hurst loz-hurst commented Sep 22, 2014

I hope that the node groups defined through this new "inventory" mechanism will be available in the pillar top.sls file too - so that those of us who solve the "what groups does this node belong to" issue with ext_pillar interfaces to other inventory systems can (finally!) match on these groups in pillar/top.sls.

@mumblez
Copy link

@mumblez mumblez commented Oct 3, 2014

+1

3 similar comments
@HontoNoRoger
Copy link

@HontoNoRoger HontoNoRoger commented Jan 29, 2015

+1

@Galser
Copy link

@Galser Galser commented Mar 24, 2015

+1

@dharmab
Copy link

@dharmab dharmab commented Jul 31, 2015

+1

@stale
Copy link

@stale stale bot commented Dec 18, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

@stale stale bot added the stale label Dec 18, 2017
@loz-hurst
Copy link
Contributor

@loz-hurst loz-hurst commented Dec 19, 2017

We now do this with ext_pillar to import "group" pillar information based on a custom module that reads the data from a combination of external inventory systems. The addition of ext_pillar_first (available in stable after this bug was filed) means we can target using the "group" from ext_pillar in our pillar files.

In short, for our use case we have a workaround through other developments in salt in the meantime.

@stale
Copy link

@stale stale bot commented Dec 19, 2017

Thank you for updating this issue. It is no longer marked as stale.

@stale stale bot removed the stale label Dec 19, 2017
@iavael
Copy link

@iavael iavael commented Dec 19, 2017

@loz-hurst

Unfortunately it doesn't help if there are only ext_pillar sources and one of them provides "groups" list and it's needed to use these groups to target pillars from other sources.

@loz-hurst
Copy link
Contributor

@loz-hurst loz-hurst commented Dec 20, 2017

@iavael Indeed, which is why I said "for our use case". It's very much a workaround, not a fix, as we've had to create a single ext_pillar that interfaces with the other ext_pillars (and some sources that we've not bothered to write a proper ext_pillar interface around, since we can't use it direct because of this) so we can cross-reference and cross-use them before using (some of) the information in them to target the pillar_root files.

The functionality to do it properly in salt would be much better than mangling ext_pillar sources via our own hacky interface to get it to work.

@stale
Copy link

@stale stale bot commented Apr 14, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

@stale stale bot added the stale label Apr 14, 2019
@iavael
Copy link

@iavael iavael commented Apr 18, 2019

As a workaround we currently target pillars to nodegroups in the external system. Basically, our internal service that provides nodegroups for state targeting via pillars, also began to provide some actual pillar data that was stored in git.
This solves some of the most painful moments (at least we got rid of bunch of cryptic regexps in pillar top.sls), but this is very kludgy, because targeting mechanism is now outside of saltstack.

@stale
Copy link

@stale stale bot commented Apr 18, 2019

Thank you for updating this issue. It is no longer marked as stale.

@stale stale bot removed the stale label Apr 18, 2019
@sagetherage sagetherage removed the P1 label Apr 21, 2020
@bbinet
Copy link
Contributor

@bbinet bbinet commented May 6, 2020

Any news on this issue?

Is it normal that I'm not able to target minions using pillar (salt -I or salt -J) when my pillar data sit exclusively in reclass ext_pillar ?
I don't understand why pillar data coming from ext_pillar could not be used to target minions like any other pillar data? shouldn't it be considered as a bug?

@iavael
Copy link

@iavael iavael commented May 6, 2020

@bbinet you can target jobs (via salt -I) and states (via top.sls) by pillar data (including ext_pillar). We successfully ise this in production for several years. You cannot target pillar via top.sls by pillar data.

@bbinet
Copy link
Contributor

@bbinet bbinet commented May 6, 2020

@iavael it does not work for me for all my minions (version 2019.2.0), but I've just come across issue #52234 and I think I should upgrade my minions to latest 2019.2.4 which should have the fix.
I've just tried to target an old minion (2018.3.3) with salt -I and it does work!
So I think I just need to upgrade, thanks.

@iavael
Copy link

@iavael iavael commented May 6, 2020

@bbinet that's strange. We used such targeting since 2017.7. It could be available even earlier, but we didn't use saltstack then yet.
But if it works for you after upgrade, then it's fine, I suppose.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.