Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
node lists support implemented #9
Idea of this patch is to create special file, which denotes list of nodes, and by design,
example: (also example available in examples directory)
we can create file group1.hosts and group1.yml, thats gives us effect of several identical .yml
as for performance, seems that it works reasonably fast, event for 4k hosts.
additionally, this patch fixes bug with inner directories. If we create node definition in
$ ./reclass.py -b examples -n ttt
Would a mapping of nodenames to classes help? Like this: http://reclass.pantsfullofunix.net/todo.html#wildcards-regexpclass-mapping
Can you tell me, why proposed solution is not fit reclass? It don't brake any compatibility, but gives huge flexibility. Even host generators can be implemented in that case (we can easily implement support for generators like some.host[1-30].domain in hosts files)
I am not saying it's not acceptable, I just wanted to understand your use-case before I look at the code!
Just to confirm that I understand correctly what you want: if matching were implemented, this would also have the same effect, right:
That is, matching each node name explicitly and assigning a group.
Could you file an issue about the error you found? I would like to fix this separately.
Solution with matchers not solves completely:
Btw, where this matches should be defined? In class or nodes .yml files?
referenced this pull request
Nov 26, 2013
How would you feel about adding a new top-level key to node YAML files, e.g.:
Furthermore, the parameters
I've given this issue some thought today while out on a walk.
What we are trying to achieve is to treat several nodes equally, i.e.
However, this is not enough, as run across the entire inventory need to be
So somewhere, we need to keep a list of nodes.
There are four proposed ideas:
If we can agree that a directory of empty files is the same as a file listing
(3.) is identical in functionality to (2.), except it combines two files into
(1.) and (4.) are identical, if the node files are created with a common
(2.) and (1.)/(4.) are really not any different, because whether
So there's really only (1.) and (4.). Let's compare those a bit more.
Updates to (4.) are atomic, we had that. (4.) is mostly implemented. What's
(1.), OTOH, would allow for lines such as
However, this is common shell parlance, and
I am therefore tending towards (4.), and I would like to invite you to take
What do you think?
Schemes (1), (3) and (4) has one significant drawback: it is not possible to guarantee order of class applications, and scheme (4) doesn't guratatee application at all.
As for your assumption, that we have www0.example.org through www999.example.org without holes is not really true (especially in case, of disabled nodes for maintenance or some experimental classes application).
And we don't need to restrict user to use some ephemeral node name (actually a group name), and even allow reclass to return information about group as a whole (by requesting group by its name).
As for (4), it depend on node names, in my case it is nearly impossible to control node names.
I don't understand your point about order of class application. It would be well-defined: if a node
If you have holes in a sequence, i.e. because nodes are disabled, then scheme (4.) makes this easy: instead of empty file
and it will receive the
I also don't understand your "host file atomicity" comment. Writing/changing files on Unix is not atomic. Adding and removing files to/from a directory is atomic. Can you please try to help me understand what you are saying?
"whole class/properties assignment to whole group." is also possible with scheme (4.), you just have to go via a class, which makes a lot of sense.
If you have "a pool of machines (say host[0-100].some.domain) and those hosts can be tossed between different roles or even clusters", then scheme (4.) would work just fine: if a host changes role, you just
I am sorry, I am really not understand your problems with (4.). Could you maybe make some examples, after you look at class mappings?
"I also don't understand your "host file atomicity" comment. Writing/changing files on Unix is not atomic. Adding and removing files to/from a directory is atomic. Can you please try to help me understand what you are saying?"
"I don't understand your point about order of class application.", Ok, I've looked at your branch, looks like I need to manage config for reclass and carefully edit mapping. And this solution still use matching, so in degenerate case, I need to put all of my hosts in one config file. It gives complicated solution, instead of clean and easy management of simple files or list of aliases in node definition.
""whole class/properties assignment to whole group." is also possible with scheme (4.), you just have to go via a class, which makes a lot of sense."
"If you have "a pool of machines (say host[0-100].some.domain) and those hosts can be tossed between different roles or even clusters", then scheme (4.) would work just fine: if a host changes role, you just mv it to a different directory, which is truly atomic."
With my patch it works like a charm: I dump group memebers to group.hosts file, dump role definition to group.yml and even if I need atomicity, I can simply copy whole inventory and switch it with two move (and can be supported by reclass if needed by say special .lock file). Classes are merged from different salt states git repos (where actuall states defined and default classes defined too). Nodes can override some settings
As result I have installation, which has no parts, which need to be managed by hand, thats gives me ability to construct multimaster scheme, where all hosts only reads confiugation from elsewhere, and not store anything, what need to be repliacated to other masters.
It sounds to me more like you need a different storage backend, yaml_fs is really not made for automated use. Wouldn't an LDAP backend work for you?
Other question: if nodes in a subdirectory would always get assigned the class named like the subdirectory (without mappings), this would work for you, right? I.e. a list of files in
"It sounds to me more like you need a different storage backend, yaml_fs is really not made for automated use. Wouldn't an LDAP backend work for you?"
Don't understand, why .hosts solution don't fit. It can be implemented like me in this pull request, or like you suggested via attribute in node definition, but it much simplier then any other solution:
Really, really don't understand, why you decline so simple solution like hostlist and subdirectory scanning, like done by this patch.
"Btw, I agree with you that the class mappings should not really be put into the configuration file. That's not final, but it was convenient to start with."
The classes applied with mappings are in well-defined order. If you think otherwise, I suggest you provide a working case proving me wrong.
One reason why I am against
And the reason why I am not so keen on
I understand your concerns about class mappings, but I think they are they way to go. We just have to come up with a proper way to define them.
If you have several sources of class mappings, you should define how they applied to nodes. Now you have one source - config. But if sources will be two?
Why you need so purification and so many details on upper level? In case of another storage implementation it can be many optimization on storage level, how to store groups. For example if we have, say, SQL storage, we can use 1 sql query to get all hosts for node, but if you don't allow implementation details, reclass will ask all nodes one by one.
And what you mean 'administering like yaml filesystem'? what that mean? abstraction - one node one file breaks with class mappings and make administering much much harder, then hosts file or node mappings.
But it is ok, you have you vision of how you system should look like, but making wide interface to filesystem is really bad idea, because it limits implementation (right like in this pull request, you current patches not compatible at all, because they pull details and caching to upper level, and storage has no chances to hint upper level about nodes: node1 and node2 actually the same nodes and you don't need to ask storage about them once more)
Either there will be only one source of class mappings, or the sources will themselves be ordered. Reclass will always have well-defined, reproducible output.
An SQL storage backend can run a single query if that's how it's implemented, just like the new yaml_fs enumerates the inventory only once and stores the listing in a cache.
At the moment, there is only a
yaml_fs is going to stay like it is.
But if another storage backend comes around and the interface between the "upper level" and the storage backend isn't suitable, then it will evolve.
I would accept a patch that
The idea is to parse