[catalog-next] on staging

[adborden]

GSA/datagov-ckan-multi#296

This makes a few significant changes.

• Refactor groups for catalog-next
• Sandbox now uses a static inventory
• ansible-inventory-diff for comparing inventory changes
• debug.yml playbook to check individual variable resolution

catalog-next Ansible groups

The biggest change is the refactoring of catalog-next groups for the Ansible
inventory. I was having a lot of trouble reasoning about the different roles
within catalog/catalog-next, so catalog-next groups are very different than
catalog classic. I left catalog classic as-is.

Groups now follow a tree structure within an app (catalog-next) and then have
specializations and sub-specializations. Variable interhitence works with each
child group overriding some variables from the parent. Ansible isn't really
meant to work this way, so this feels kind of like a hack, but at least it's
predictable and easier to reason about.

$ ansible-inventory -i inventories/staging --graph catalog-next
@catalog-next:
  |--@catalog-next-web:
  |  |--@catalog-next-web-a:
  |  |  |--catalogweb1d.dev-ocsit.bsp.gsa.gov
  |  |--@catalog-next-web-admin:
  |  |  |--catalogpubweb1d.dev-ocsit.bsp.gsa.gov
  |  |--@catalog-next-web-b:
  |  |  |--catalogweb2d.dev-ocsit.bsp.gsa.gov
  |--@catalog-next-worker:
  |  |--@catalog-next-worker-main:
  |  |  |--catalogharvester1d.dev-ocsit.bsp.gsa.gov
  |  |--@catalog-next-worker-misc:
  |  |  |--catalogharvester2d.dev-ocsit.bsp.gsa.gov
  |  |--@catalog-next-worker-qa:

Sandbox static inventory

Sandbox was using a dynamic
inventory
to fetch hosts from the AWS API at runtime. Since changes to Sandbox are made
via Terraform, in theory, Ansible can dynamically pick up those changes without
any change to the hosts file. However in practice, we often had to tweak groups
in sandbox/hosts to match Terraform and when a conflict arose, it was painful
to resolve it.

Additionally, you need access to the AWS API to use ansible-inventory, which
I found annoying, and contributes to environment drift with the BSP
environments (we should be able to use the same development tools in all
environments).

This PR removes the dynamic inventory and uses a static hosts file like the
other environments. If a new host is added via Terraform, or a hostname
changes, we'll have to update inventories/sandbox/hosts. We can now lint the
sandbox inventory, like staging/produciton and this also enables us to use
ansible-inventory without AWS credentials and the tools/tricks described
below.

ansible-inventory-diff tool

One gotcha with Ansible is that it's easy to change a variable in a way that
has a much broader effect than intended. I've often used ansible-inventory --host $host to manually inspect variables. However, what I really want is a
way to diff the ansible-inventory output.

That is what ansible-inventory-diff hopes to be. This is a work in progress
and there are probably a few bugs to work out. ansible-inventory-diff will
generate what I call an "inventory manifest" from the current working directory
then generate a second inventory manifest from another git commit and then
diff those inventory manifests. The inventory manifest is just a giant JSON
output of all of our environments' ansible-inventory output, including all
variables and vault secrets.

$ bin/ansible-inventory-diff origin/develop

You'll get a diff against origin/develop in your favorite difftool.

What else can we do with this? We can use this in CI to enforce that any
changes to the inventory have been explicitly reviewed by a developer, similar
to how we review the Ansible vault locally when reviewing PRs.

How? We could hash the inventory manifest and commit that hash to git. In CI,
we can compute the inventory hash and compare it with the known hash. If they
match, all good. If they don't, it means something in the inventory has changed
and a human should manually review the inventory manifest to make sure the
change was intented. Assuming the change is correct, they can update the
inventory manifest hash.

Anyway, this is pretty early, but I found this useful as-is for refactoring
inventory variables so I wanted to share.

debug.yml playbook

This playbook can be used in local develepment to check the resolution for a
single variable for a specific host in a specific inventory.

$ ansible-playbook debug.yml -i inventories/staging -e debug_variable=catalog_db_user --limit catalogharvester1d.dev-ocsit.bsp.gsa.gov

PLAY [Debug] ******************************************************************

TASK [debug variable] *********************************************************
ok: [catalogharvester1d.dev-ocsit.bsp.gsa.gov -> localhost] => {
    "msg": "catalog_db_user=ckan"
}

PLAY RECAP ********************************************************************
catalogharvester1d.dev-ocsit.bsp.gsa.gov : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

[adborden]

I think this is ready for review. We're still waiting on BSP to install the certificates, so I've set the catalog-next hosts to datagov_in_service: false which makes this PR merge-able. Once the TLS certs are installed, we can re-enable them in another PR.

[jbrown-xentity]

Major changes, but did a deep dive and it all looks good to me!

[adborden]

👍 yes, it's kinda big. I'm happy to walk through this with anyone else if they'd like.

[adborden]

note to self: add a comment about inventory catalog-next vars won't override these.

[woodt]

This looks quite reasonable and definitely an improvement.