Fixes improvements for docker compose and terraform quickstarts

[douglaswainer]

Description

The primary purpose of this PR is to make the docker-compose quickstart easier to get working out of the box for people trying to run it on bare metal machines with NVME drives. It also addresses the some issues I encountered when testing these changes with the other docker-compose based quickstart guides.

• Docker-compose (on bare metal)
• Terraform Equinix metal
• Vagrant (with USE_HELM=false

Plus some cleanup to remove some code/files that I believe are legacy and no longer used.

Why is this needed

This is my attempt to address 172 and 173.

How Has This Been Tested?

I've tested by running through the quickstart documentation for all compose based quickstarts:

• Docker-compose (On bare metal Lenovo Thinkcenter Tiny M900 machines, with NVME drives)
• Terraform
• Vagrant/Libvirt with USE_HELM=true
• Vagrant/Libvirt with USE_HELM=false

How are existing users impacted? What migration steps/scripts do we need?

All statements like {{ index .Hardware.Disks 0 }} in the deploy/stack/compose/manifests/template.yaml have been replaced with environment variables (set in the .env file) that get substituted by the manifest-update container task. This should make it more likely to work for new users, but might confuse users that add their own manifest files.

Checklist:

I have:

• updated the documentation and/or roadmap (if required)
• added unit or e2e tests
• provided instructions on how to upgrade

[douglaswainer]

I've been looking into the hardware types and I can see how I could have used the MetadataInstanceStorageMount, which I believe could be substituted more naturally (or more inline with Tinkerbell templates) with:

{{ (index .Hardware.Metadata.Instance.Storage.Filesystems 0).Mount.Device }}

Provided the hardware.yaml provides this metadata. I might be wrong, I'm not very familiar with Golang templates so I'll need to test.

I'm happy to close this and go and rework/retest my changes and then reopen again at a later stage, if that is preferable?

[jacobweinstock]

Tremendous work, @douglaswainer, thank you!

[jacobweinstock]

I've been looking into the hardware types and I can see how I could have used the MetadataInstanceStorageMount, which I believe could be substituted more naturally (or more inline with Tinkerbell templates) with:

{{ (index .Hardware.Metadata.Instance.Storage.Filesystems 0).Mount.Device }}

Provided the hardware.yaml provides this metadata. I might be wrong, I'm not very familiar with Golang templates so I'll need to test.

I'm happy to close this and go and rework/retest my changes and then reopen again at a later stage, if that is preferable?

Hey @douglaswainer, at the moment only the disks (hardware.Spec.Disks) are available for use like this. This was deliberate on our part so that we could evolve this functionality with purpose as use-cases arise. So no need to rework this. Thanks though!

[douglaswainer]

Hey @douglaswainer, at the moment only the disks (hardware.Spec.Disks) are available for use like this. This was deliberate on our part so that we could evolve this functionality with purpose as use-cases arise. So no need to rework this. Thanks though!

Oh interesting, I assumed the whole hardware data json could be referenced. Thank you for clarifying!

[jacobweinstock]

im not too excited about duplicate .env files to maintain. any chance we could have only one?

[douglaswainer]

Yeah I'm not happy about it either, I can look into options for merging them into one but I don't know of an easy solution based on .env files.

There's a few bits of reused variables and bash code in the project, I was thinking of converting the bash provisioning to Ansible and splitting out the more specific provisioning tasks with roles based on whether the machine is bare metal, equinix, vagrant etc.. Though I think that would be a separate issue/PR for later on.

[jacobweinstock]

thanks for the response and especially for your work and patience on this!