[WIP] Refactoring configuration mechanism

[jharmison-redhat]

Rationale

The configuration mechanism is currently a curses-based Terminal UI. This is difficult to instrument with more automation, and is difficult to navigate in some circumstances. A more intuitive configuration mechanism is desired. Because this affects a large portion of how many components work, it will be a sweeping change.

This WIP PR should not be merged, and is used to demonstrate progress towards these goals to facilitate a conversation around the code written to satisfy them.

Occasionally during development and before merging, interactive rebases will be performed that squash like commits into chunks oriented towards each goal.

Progress

• Implement mechanism for doing drive detection that is less interactive and more portable/parsable
• Define new YAML structure for overall configuration
• Validate that configuration changes don't break existing inventory variables
• Change wipe and container-storage deployments to read and parse new configuration, rather than prompting via partial TUI implementation
• Read and format configuration, either in dedicated library or in embedded inventory ( fixes Create Python library for reading config  #100 , fixes dict editor box with currupted entries #83 )
• Document configuration format, add example config
• Create a form to generate a configuration file with hand rails ( fixes More intuitive configuration mechanism #76 )
  • Ensure form doesn't show data for drives detected as having a root filesystem on them ( fixes dont show root drives in wipefs and ocs deployments #77 )
• Import and export mechanism for configuration files ( fixes farosctl import and farosctl export commands for managing config #103 , fixes backup and restore bastion node config #35 )
• Remove remaining environment variable references ( fixes Remove all env var references #101 )
• Remove config TUI ( fixes remove config tui #102 )

[jharmison-redhat]

I think the right workflow now is to implement the configuration form, probably as a container running the web app so it's easily portable later but for now with userns'd podman as the bastion user. Then the farosctl command to start that configuration image can first start the nodes, interrogate the drives, and then present the appropriate subset of them to the user for selection along with the rest of the configuration - then the drive list can get templated into the YAML and added to the inventory along with the rest of the configuration.

Because I don't know yet if that's the best answer for configuration workflow, I can't really implement the wipefs and container-storage fixes to use that new configuration.

I'm going to focus next on building a very simple web app for templating the configuration in. I'd also like to move the shim variables into templated config and stop reading them at runtime every time - which means we can read them from the environment when the config application is started, then pass them to cluster-manager via the config yaml.

[jharmison-redhat]

Also, the state of the config right now is that it's templated into inventory variables 1:1 the same as master, with a couple of exceptions:

• subnet_mask is an integer, instead of a string. I'm not in love with keeping the CIDR prefix as a separate item, since Ansible is ipaddress aware, and this will probably be refactored away as propagating the changes will result in the ability to adjust the playbooks/roles to use the proper ipaddress types.
• ignored_macs is a proper list of dicts, with mac and name keys for each item, instead of a string of json. I haven't looked at the code that uses this variable, but I imagine it's probably doing |from_json right now, and the propagation of changes should allow us to simply work with the variable.

[rmkraus]

Notes from today's review:

• Config webapp to be hosted on heroku
• webapp installed in cluster-manager
• multiple config versions available on hosted app
• migrate config versions on farosctl update
• config in its own repo
• schema changes to config yaml to track version and kind
• delay structure propogation until later release
• container storage later, possibly new config kind?

[jharmison-redhat]

I have updated the PR with a commit that strips configuration entirely from this repo in support of the config in its own repo point.

I anticipate that there may one or more commits to this PR directly in support of the updated goal of verifying that the existing playbooks and roles react to the small changes mentioned in my earlier comment. I adjusted the goals to remove the propagation of structured variables through the roles/playbooks for now.

There should also be one more commit to this PR directly to bump the pinned version number of the faros-config package, which is marked here.

Additionally, I moved the faros-config specific goals from the above discussion notes into faros-config issues.