Generate OpenStack Ansible inventory

[jimbaker]

Script/corresponding common library code using underlying SQLAlchemy model representation that generates Ansible inventory files that's suitable for using with OpenStack Ansible.

[nrb]

I've put a few months of work in the openstack-ansible dynamic_inventory.py file. Would you like some help with this?

[nrb]

In particular, I'm not seeing that Craton has concepts that map to OpenStack-Ansible's environment structures, which describes how various service containers map to hosts. Is this in scope for the project?

http://docs.openstack.org/developer/openstack-ansible/install-guide/app-custom-layouts.html has information on how the current environment configuration works, and http://docs.openstack.org/developer/openstack-ansible/developer-docs/inventory.html has a high level overview of how we already generate our inventory.

[sulochan]

@nrb any help from OSA team is welcome :) I've had a few chat with @odyssey4me from OSA team re the points you raise above and we are meeting early next week to clear these points out. Perhaps we should have a joint convo on this ?

re: mapping to OSA env structure, yes it in the scope in a more generic term. We support labeling host .. and are looking at labels support hierarchies etc, which will map host->service or any obj -> other obj. I think this will work for OSA env as well (hopefully for as is). The part that I mostly want to solve right now is to make sure we can generate all the host groups you guys use (which is a lot ;) ) in generic enough way that it work for OSA but also for any other ansible inventory. Same applies to vars / hostvars ... such that OSA can choose between using the file based structure you have now or switch toward craton.

The plugin based approach work you guys started looks particular interesting at that point:
https://review.openstack.org/#/c/287586/1
https://review.openstack.org/#/c/269236/

In any case lets have a chat next week and go from there.

[sulochan]

Also see: #110

[odyssey4me]

@sulochan @jimbaker I think we should do a kick off chat to discuss next steps. I think we need to prepare to have some sort of testable inventory usage from Craton for OpenStack-Ansible in the Ocata cycle. Ideally the kick-off should include both @nrb (US-based) and @neillc (AUS-based) but the time zone co-ordination may be difficult.

Fundamentally I'd like to be clear that instead of having Craton generate the files for an OSA inventory, we would rather implement a pluggable library into the OSA dynamic inventory that can be configured to use the files (for gate tests and small environment testing) or to use Craton (or any other inventory system given the appropriate plugin). OSA should implicitly get all host/group details (including container names and addresses) and all configuration that overrides defaults from the inventory system.

This may require implementing some tooling to 'import' required default host groups, to 'generate' container host entries, and also to implement IP address allocation for the containers (perhaps via a plugin to an IPAM system). These should be actions that are only done on occasion (probably before major upgrades). These are details we need to hash out though. In principle I'd like to ensure that instead of generating flat files based on data from Craton, that we dynamically read the data from Craton and OSA should simply trust Craton as the inventory with all the information it needs to deploy an environment.

[nrb]

@sulochan So this would require deployers to define that structure manually for each deployment? It seems that there would need to be some way to provide Craton with a hierarchy in such a way that users don't have to enter the same thing every time; I would think some kind of import as mentioned above would be a solution.

The linked reviews were indeed the start of a pluggable system, though I think they'll need to be revisited due to the age and some other code clean up changes that happened to the dynamic_inventory.

[sulochan]

@nrb yeah absolutely agree. idea is to come up with templates such that users can easily put this data into the system without repeat. We had some chat re: inventory and how to make it easy for end users to push OSA required data (for new and for existing OSA deploys) and consume it back etc.

This is what the thinking is so far:

From high level, Craton will provide the following:

1. /ansibleinventory?region=dfw -> OSA specific inventory data for ansible. This is the same json data that you'd get if you ran your inventory generation file today. It will have sorted out all the groups, vars, hostvar etc as OSA expects it. This is driven by labels, i.e: you tag a host with service label(s) and each service found in labels will be turned into a group. This helps preserve linking of groups such that containers -> host -> cell -> region. (maybe cabs / network switches etc also)

2. /ansibleinventory?region=dfw&head=-1 -> same as above but gives you, one iteration back. (Need to discuss and finalize how we save iterations. Does every change save as an iteration, or should we have the ability to save it manually etc.)

3. for empty groups that is not yet defined but expected by OSA, OSA will need to assume empty dict. There is no way for Craton to know this.

4. Provide templates to push data into Craton. For example, the initial work to generate inventory for OSA deploy should be made as easy as possible by both providing the generic template to create this with few info as possible, but also be able to make as much change as needed. This should reflect if someone was to do this thought the file based system now for example.

5. Provide scripts to be able to consume existing OSA deploy data (inventory + config scripts) and push that into Craton such that dependency on the files is removed.

OSA can then use teh pluggable system discussed above to either point to files (existing) or towards craton. Using craton should provide more functionality such as:
a) using craton workflow to make audits and populate inventory. One example of requirement from OSA team is to audit hosts to get host vars (what ansible would collect if you run it with gather_facts) and populate that etc.
b) Tie in workflow to do deploys and upgrade.
c) Tie in workflow to do auto remediation of alerts
...
etc

[sulochan]

• https://etherpad.openstack.org/p/openstack-ansible-newton-dynamic-inventory

Is the OSA inventory discussion, someone passed this to me and it has a lot of info that is relevant to this discussion. Note: some of the req's there is already implemented or being implemented currently.

[evrardjp]

For the point 3) above, this is not the expected ansible behavior, but it's indeed OSA behavior. So I guess we could workaround that. I don't think it should be part of craton's scope anyway, because OSA can have a merged view from different sources, and define what we need, independently of craton.

For the a) above, I'll maybe rephrase what I said in a previous meeting: ansible can handle its own collection of facts, or re-use a facts collector. We just need to know what we can/should trust as fact collector, and make appropriate design decisions.

I think there is something missing right now from the list of features, is constraints per variable type.
For example, we are currently saying "for container x pick an IP from this pool, excluding these IPs. don't pick an existing IP.". With craton, we would just trust craton's output.
To make sure every deploy would work fine, we'd have to ensure that IPs generated by craton for the containers are, well, unique. This is a constraint that will have to come from the inventory, and that we won't be able to check, unless we edit the dynamic inventory lib. Would that be possible to introduce this in craton?

[nrb]

1. /ansibleinventory?region=dfw -> OSA specific inventory data for ansible. This is the same json data that you'd get if you ran your inventory generation file today. It will have sorted out all the groups, vars, hostvar etc as OSA expects it. This is driven by labels, i.e: you tag a host with service label(s) and each service found in labels will be turned into a group. This helps preserve linking of groups such that containers -> host -> cell -> region. (maybe cabs / network switches etc also)

In OSA's current model, not all hosts are known at the start. The dynamic inventory script creates the container names/IP addresses and inserts them into the inventory so that LXC playbooks and roles can create them. This could be modified to send that information to Craton first, though.

This is similar to @evrardjp's concern - is it within Craton's scope to actually generate this information? Or is Craton only meant to store? I know many OSA contributors would like to completely hand off the generation, but I'm not so sure it can be done cleanly, since any system that would do such a thing must be aware of said constraints somehow, so OSA would at least have to provide those.

Along the same lines of the used_ips variable, we have two other "global" lists/dictionaries that don't apply directly to one host: cidr_networks, which defines the available network cidrs and their names for all hosts, and global_overrides, which currently includes mostly container interface configuration information, as well as some swift overrides.

Kind of thinking aloud as I type, but I'm assuming these would be assigned to group variables outside of Craton? These are, again, mostly used in container creation, which is currently done via LXC playbooks and roles, not with a centralized system.

[nrb]

A question about this assumption

/ansibleinventory?region=dfw

In private cloud scenarios, especially many currently deployed, there's only one region. I'm going to assume Craton itself doesn't care what it would be called, but OSA may want to decide on a generic region name to use for single-region installs.

[sulochan]

In OSA's current model, not all hosts are known at the start. The dynamic inventory script creates the container names/IP addresses and inserts them into the inventory so that LXC playbooks and roles can create them. This could be modified to send that information to Craton first, though.

@nrb this is the exact case i discussed with @odyssey4me So in general craton inventory is only supposed to store. There is nothing in the inventory itself that does this, BUT, the template that i was talking about previously, is a client side code, that will allow to do such things .. for instance we will only ask for the min required subset of things. For example the cidr range to use for your container networks, then if you choose the default template .. container name, ip assignment etc will be auto generated for all containerized services from the list of hosts available (assuming the hosts have been pushed into the system at this point), and user and obviously choose to put this info manually. Some of the exact implementation details will need to be hashed out though. But whats your thoughts on something like this ?

[anguslees]

On Thu, 7 Jul 2016 at 01:34 Sulochan Acharya notifications@github.com
wrote:

From high level, Craton will provide the following:

1. /ansibleinventory?region=dfw -> OSA specific inventory data for ansible.

[and similar]

Fwiw, I recommend we generate the ansible inventory on the REST client
side, rather than in the craton server:

• ensures our REST API is expressive enough to support this sort of
  use-case.
• avoids embedding ansible-specific knowledge in the craton server, and
  hence needing a heavyweight plugin architecture for other deployment tools.
• ensures host/attribute ACLs will be honoured, without requiring us to be
  diligent while coding a second access path to the same information.
• means we can iterate OSA and craton through major version changes without
  tying the two together.
  • Gus

[anguslees]

On Fri, 1 Jul 2016 at 21:45 Jesse Pretorius notifications@github.com
wrote:

Fundamentally I'd like to be clear that instead of having Craton generate
the files for an OSA inventory, we would rather implement a pluggable
library into the OSA dynamic inventory that can be configured to use the
files (for gate tests and small environment testing) or to use Craton (or
any other inventory system given the appropriate plugin). OSA should
implicitly get all host/group details (including container names and
addresses) and all configuration that overrides defaults from the inventory
system.

Does a direct-craton-query version introduce (larger) consistency problems?

Specifically, if a long OSA run takes (eg) a day to complete, then that's a
big window in which someone might change a hostname/whatever value in
craton. It would be bad if some of the OSA run saw the old value, and some
saw the new value.

We still have a small version of this problem when generating files, but
presumably the file generation is quick - and then the long OSA run can
complete, be retried, etc using a fixed and self-consistent snapshot of the
craton inventory.

• Gus

[sulochan]

Fwiw, I recommend we generate the ansible inventory on the REST client
side, rather than in the craton server:

• ensures our REST API is expressive enough to support this sort of
  use-case.
• avoids embedding ansible-specific knowledge in the craton server, and
  hence needing a heavyweight plugin architecture for other deployment tools.
• ensures host/attribute ACLs will be honoured, without requiring us to be
  diligent while coding a second access path to the same information.
• means we can iterate OSA and craton through major version changes without
  tying the two together.
  • Gus

yeah, + 1

This is what i was suggesting initially as well. In a way we are already providing the api to do this .. i.e what is done internally is just logic on top of host, cell, region queries ... which works just the same if it were a client making that rest api call and doing that processing client side.

I think what we need to figure out is what value can we add by doing it craton side .. without being OSA specific ... i think its probably not too bad to provide ansible specific inventory endpoint but we definitely should not get tied to OSA and vice versa such that one needs a change in the other code base to work.

[nrb]

Specifically, if a long OSA run takes (eg) a day to complete, then that's a big window in which someone might change a hostname/whatever value in craton. It would be bad if some of the OSA run saw the old value, and some saw the new value.

In the current system, only the IP addresses and hostnames for containers are new values inserted into the inventory. Everything else should be variables in YAML formatted into JSON. So, there does exist the possibility of editing files by hand during deploy, but the files are ingested exactly once per run,

I agree with not hard-coupling either project to each other. However, there are certain constraints that the OSA system expects, such as generating IP addresses that aren't already in use. My sense is that the core OSA team would like to move this functionality out of the OSA repository, but I don' tthink Craton is the right place for it, either. More likely, some sort of client-side code would do this work and send it to Craton.

I'm of the opinion that should be in the OSA side. I'm not enthusiastic about creating another project just to handle that part.