WIP: Cloud upload

[evan-goode]

Thought now would be an appropriate time to start a conversation about the progress on cloud upload so far. I'll start by describing the high-level design and then lay out what I think still needs to be done.

I've managed to keep the bulk of the upload logic separate from the rest of lorax. It currently exists as a package called "lifted" in src/lifted. Uploads are queued and stored in a way similar to how composes are. When an upload is scheduled, an Upload object is created with the upload's settings and state, and this object gets serialized and written to /var/lib/lorax/upload/queue/$upload_uuid. If an upload is scheduled from a compose that isn't finished yet, the upload starts with a status of "WAITING". As soon as the associated compose finishes, the upload gets marked "READY". A monitor thread watches for "READY" uploads, kicks each one off as separate process, and marks them as "RUNNING". Since each running upload gets its own PID, we can cancel an upload simply by sending it a SIGINT.

The actual uploading is done using Ansible playbooks and the Ansible cloud modules. I currently have uploads to Azure, vSphere, and OpenStack working. Each cloud provider has both a provider.toml file containing some metadata (mostly describing the credentials and settings it expects) and a playbook.yaml. When a provider's settings get saved for future use, they're written to /var/lib/lorax/upload/settings/$provider_name.toml.

The API should now support most of what we'll need on the front end, including:

• Scheduling an upload to run as soon as a compose finishes, or immediately from a finished compose
• Cancelling uploads
• Retrying a failed upload with new settings
• Fetching an upload's Ansible logs
• Validating, storing, and retrieving settings per cloud provider

Some things we still need to work on:

• Gracefully handling missing dependencies: any upload needs python3-ansible-runner installed, and right now lorax-composer just crashes if it's missing. Furthermore, each cloud provider has its own set of dependencies. API routes should return some special error if any required package is missing, and Cockpit would ideally offer a one-click install.
• The Ansible cloud modules themselves need a little work:
  • The Azure modules depend on an older version of the Azure Python SDK than is packaged in Fedora.
  • The AWS modules are missing some functionality we need; see Allow creating EBS volumes/snaphots from s3 files ansible/ansible#53453. I'm not sure the solution mentioned there addresses our problem.
  • We'll need to write a couple new modules if we want to support Alibaba Cloud. There isn't anything yet for uploading or importing VM images.
• Configuration: I just tacked on an "upload" section to the composer config, so let me know if you think there's a more appropriate way to pass paths to lifted.
• Scheduling and managing uploads from composer-cli
• Uploads should be run as an unprivileged user
• Logs should probably not contain sensitive API tokens

For now, the following installations should be all you need to get Azure, OpenStack, and vSphere working:

sudo dnf install -y python3-ansible-runner python3-openstacksdk
sudo pip3 install ansible[azure]

[bcl]

Great job! This is looking really nice. Here's my list of comments:

• Have just a provider name isn't enough, there will be cases where users will need multiple accounts per-provider.
• I don't like pickle on disk, TOML is what we're using everywhere else -- it makes it way easier to debug things when they get stuck/broken
• When deleting a compose it needs to look for a running upload and return a new error
• style nitpick: LOG should be log like we use in the other modules.
• Do we want to include the upload logs for a compose in that compose's log directory? Might be nice to have it all in one place so you can download it with the metadata API route
• The image types shouldn't be limited to one provider. A default makes sense, but it should be possible to override and upload qcow2 someplace else.
• When removing an upload it should check to make sure it's not running (and whatever other variations on this theme, I probably didn't spot them all)

It also feels like these new API routes would be a good candidate for a v1 API instead of adding them to v0, which would also make it easier for the front-ends to figure out what's supported.

[bcl]

This is looking pretty good! I think all the pieces are now there, it just needs some cleanup:

• squash related commits, with descriptive commit messages
• split out commits like the change to the toml module into their own.
• Either add my API v1 commit from pr Add id field to source and change name to use the repo.name instead of repo.id #699 or rebase on top of it and move the new API calls into there. Or you could review that PR and I could push it :)
• Add full docstrings to the new functions.
• Tests. Unit tests where relevant, as well as cloud upload tests, maybe replacing the current ones? See what @atodorov thinks.

[evan-goode]

Great, thanks! I'll just rebase on top of your v1 branch.

[bcl]

See PR #843 for continued work on this.