Spin up an instance and run a simple test after cloud image upload

[dtgay]

Adding an idea from @mattdm:

After cloud image is uploaded:

• a test instance is booted in any one of our available environments (EC2, Fedora cloud, whatever is easiest)
• a simple test is run against that instance
  • connect via ssh and run /bin/true
  • it should be easy to sub in any other "test command" (more complicated tests can come later)
• results of that test are reported on fedmsg (and possibly by email to cloud-sig list)
• make sure test instance has been terminated and its resources released

[mattdm]

Also, I left this unsaid, but of course the last step is "terminate test instance and make sure its resources are released". Just for the record. :)

[dtgay]

+1, added to the list. As I start programmatically doing EC2 things, I'll be keeping a close eye on the AWS control panel to ensure that our resources aren't wasted.

[dtgay]

@mattdm, we just want the image to be tested by any provider? So we don't need a test run for an image by every provider? We can just use say a spun-up EC2 test instance to test an image once? Just making sure we don't want each provider to spin up its own test instance to test each image build, if that makes sense.

[mattdm]

We'll need to some tests for every provider -- especially the "does it boot" one, but also metadata services and probably some of the other basic "does it work" tests.

On the other hand, things like "are the dependencies correct", "is the image size not crazy", and other tests which aren't environment-sensitive can and probably should only be done one place.

[mattdm]

Possibly, we do the complete tests on our own openstack, and then only once that has succeeded do we bother uploading to the other services and running those tests.

[dtgay]

Alright, this is done. There are some improvements to be made, but it works. Fedimg currently does not send test results to any email list, but that could be added later if deemed not-too-spammy.

[mattdm]

Neat! If there's no email output, where can one see the results?

[dtgay]

@mattdm, a fedmsg pops out onto the bus. Topic is image.test instead of image.upload. So we have two fedmsg topics in Fedimg now. I'm unfamiliar with how things work on that mailing list, so if you think it appropriate to shoot out an email with every test success or failure, or perhaps a condensed newsletter of sorts, I can do that, too.

[dtgay]

@mattdm This is also something we can display on fedora-releng-dash

[mattdm]

A lot of the mailing lists do have automated messages like that. Maybe ask if people would like to see it there? Maybe after it's going for a while, we could have it only send failures? I know I'd like this, but maybe others won't.

[dtgay]

+1 on just emailing on failure.