How we test Portus

Miquel Sabaté Solà edited this page May 9, 2016 · 18 revisions

Tools being used

All the tests have been written with RSpec, with some extra candy so it integrates flawlessly with the other tools that we are using. The test suite is located inside the spec directory, and it can be run like this:

# Make sure to have have executed `bundle` before performing this.
$ bundle exec rspec spec

The acceptance tests are particularly slower. This is because for these tests we use the combination Capybara + Poltergeist, and it requires the database to be truncated before running each test. Integration tests are also quite slower because they have to effectively create and remove docker containers for each component for each test multiple times (because of different setups, versions, etc). Integration tests are better explained later.

After all the tests have been run, we make a last check with SimpleCov. This gem checks that the code coverage status is at 100%. This is the way in which we make sure that our test suite is as thorough as possible.

Integration testing

Integration tests are special tests that test how Portus interacts with different versions of Docker Distribution, LDAP, etc. This is all done through docker containers, and for each test we have to create & remove all the needed infrastructure, create the proper links, initialize everything, etc. This might look a bit scary, but for developers it has been all hidden inside of the spec/integration/helpers.rb file. Therefore, developers can create integration tests just like it's done with any other RSpec test, but they have to embed everything inside of a integration block (e.g. instead of a describe block).

Developers might want to setup RSpec tag to ease up the development procedure. This is all well-handled by the integration helpers.rb file, but it also accepts additional tags:

  • distribution: either a string or an array, you can specify the versions of Docker distribution to be tested. This is particularly useful if you want tests to not slow you down when developing your feature or fix. The available tags are defined in the SUPPORTED_DISTRIBUTION_VERSIONS constant inside of the helpers.rb file.
  • skip: a boolean that will tell RSpec whether this block has to be skipped altogether or not.

Moreover, the helper.rb also define useful methods like create_user, login, etc. that can help you when writing integration tests. Every method has been documented there, so it's recommended to read the helper.rb file a bit.

An example

integration "Makes sure that it works on Docker Distribution 2.4", distribution: "2.4.0" do
  it "works on 2.4" do
    name, email, password = "test", "test@example.org", "12341234"

    create_user(name, email, password, true)
    expect { login(name, password, email) }.not_to raise_error
  end
end

Besides that, when you run integration tests the integration block will automatically attach an integration tag to your block. This is useful if you need to include/exclude integration tests. For example, you can tell RSpec to run everything except integration tests like this:

$ bundle exec rspec spec --tag ~integration

Continuous integration

Every commit being pushed in the master branch (and every Pull Request against the master branch) goes through our continuous integration procedure. This consists on the following steps:

  1. Travis-CI runs the test suite and rubocop. This way, Travis-CI makes sure that all tests are passing and that the submitted code is accepted by our code style. You can find the .travis.yml file being used here.
  2. After that, the change gets sent to CodeClimate. CodeClimate checks the health of the codebase after the push has been made, and lastly it checks the code coverage of the test suite. Note that we get two reports for the code coverage: one from CodeClimate, and the other from SimpleCov. Also note that this step is skipped for Pull Requests.
  3. Last but not least, on success the results will be pushed to the Open Build Service through this script. Note that, just like the previous step, this will be skipped for Pull Requests.