Proposal: Integration Test Framework

[justenwalker]

My goal is to make it somewhat easy to create new integration tests by factoring
out the common logic into a simple framework.

It might be overkill though, maybe there is an easier way to do it.

Folder Structure

• integration_tests/fixtures - Contains folder for each test harness
• integration_tests/tests - Contains folders for each integration test

Fixtures

integration_tests/fixtures/fixture_name - Folder contains a Dockerfile
for building a Conatainerbuddy app - resulting in an image named fixture_name

The Dockerfile should specify an ENTRYPOINT/CMD and should not require any arguments.

Tests

integration_tests/tests/test_name

• Contains a test.sh script
• docker-compose.yml for setting up the test environment
• Any other resources it might need.

If test.sh returns success: 0 then the test passed, otherwise it failed

This script can make some assumptions:

• It's current directory PWD is the same as the script.
• containerbuddy_etcd and containerbuddy_consul are running and are fresh (no data)
• all fixtures in integration_tests/fixtures are created and are available as images - requiring no arguments

Running Tests

Make target integration_test

integration_test: build
  ./test.sh

The script in the top-level folder named test.sh will:

Setup Test Fixtures

• Scan through all folders in integration_tests/fixtures in alpha order
• For each fixture:
  • Copy build into integration_test/fixtures/fixture_name/build so it can easily be sourced by the Dockerfile
  • cd integration_tests/fixtures/fixture_name
  • docker build -t fixture_name .
  • Remove build folder to clean up (should add to .gitignore also)

Note: Since fixtures are created in alpha order, they can have FROM directives for previously
created images - so as to reduce duplication.

Run tests

• Scan through all folders in integration_tests/tests in alpha order
• For each test:
  • Re-run containerbuddy_etcd and containterbuddy_consul
  • Run docker compose on the docker-compose.yml to bring up the test environment
  • Run integration_test/tests/test_name/test.sh
  • Stop/kill the compose environment

[tgross]

What I might change is that it would be really nice/useful if the integration tests can use our example applications as test fixtures. This way the application that a new user might try out is one we know we've hit with testing.

I wonder if an approach that uses docker-compose (which we're using for the example apps anyway) might be useful. This way we could have a compose file that runs an example/test-target application and as one of its containers we run the test runner. If the test runner has --links to the target containers, then it'll be forced by compose to come up last, which makes the setup a little less painful.

[justenwalker]

I like the idea of combining the examples and test fixtures - since it does promote DRY. Wouldn't it make creating fixtures more difficult? Especially if you wanted to make a lot of fixtures that had just one minor change.

But I really do like the idea of using compose to set up an environment instead of having test.sh do it. How about the tests folder has a compose file which can reference any fixture_name as an image. That way, test.sh creates all of the images, and the compose files in the tests are used to set up the environment to be tested. I've updated the above proposal with the change.

TravisCI has docker-compose available.

[tgross]

Merged in #72 and awaiting 1.0 release.

[tgross]

Released in RC: https://github.com/joyent/containerbuddy/releases/tag/0.1.1