Elixir Testing

A few universal tips for testing

These tips should be language and test framework agnostic. It is definitely opinionated!

1. A test should only test what it says so in the description

Avoid testing side effects and focus on "when input A it should return Y". This encourages a coding style that focuses on reasonable and useful return values, and functions / methods with minimal side effects.

2. A test should not test too many things at once

If you follow tip 1, then a very long test description is a good indicator of a test that tests too many things at once. Another red flag is using "and" in the description.

"it creates A and B and sets the attributes and does call function C in module D"

This test can easily be split up into multiple tests.

3. Avoid being too specific with your tests

This is a tough one and needs some time to get used to, but it gets easier with experience. The problem with being too specific in a test is, it usually encourages to test implementation instead of outcome (see 4.), and makes the app harder to refactor.

A lot of the times it is sufficient to have a red-green test, "it is successful" and "it fails when given the wrong arguments".

"it assigns a string timestamp + name + id to 'identifier'"

can be more general

"it assigns the identifier string"

4. Test outcomes, not implementations

This can be harder than it sounds sometimes, and might be impossible in some unit tests. As a general tip, if the outcome includes a very specific thing, you should put the implementation for that thing into a function / method that can be tested individually. Testing outcomes, not implementation is tightly coupled with advice 3, being too specific in your tests.

As a rule of thumb, if you are repeating implementation details in your test, you are probably doing it wrong and it will make your life harder when refactoring or when specs change.

5. What to test?

Probably the hardest question to answer. There is no one size fits all answer, but generally speaking, there are a few indicators when NOT to write another test.

• View specific things that are likely to frequently change (aka "don't thest the color of a button, or the location of a button.")

• If a test repeats the implementation ("assert 1 + 2 == 3")

• Private methods / functions (should be tested in context of the higher level function calling the private method / function)

• Constant values ("assert constantA == 'This string'"), unless it is mission critical that these values match.

• Edge cases that can be avoided by "guiding the user behaviour" - you don't have to test if certain values can not be saved to the database, if the calling method / function does not allow these values. This should be tested in the higher level method / function (A calls B, A does not allow values < 0)

• Don't repeat things you already tested. If a higher level function already has tests that make sure a certain condition is satisfied, testing it in a lower level function again will make refactoring harder.

A good tip to avoid over-testing is to start with a test on the highest level, define what you expect this method / function to do, and work your way down. Don't retest things that are already tested in the higher levels and you should end up with very few tests for the low levels that concentrate on a very specific task, and a nice description of things that should happen on the higher levels.

6. TDD, London Style, Chicago Style and other paradigms

Knowing what other people think is good, but don't take anything for gospel. Take whatever advice you think is good, mix and match! Develop your own style - without disregarding what other people do.

I don't think TDD is bad, neither is London Style or Chicago Style, but I think you should not go out of your way to follow either to the point! Instead of asking "Does paradigm X allow this?", you should ask "Does this fit with this projects style, or should we introduce this?"

7. Some examples

# Example 1
#
# This does not do what the description says and tests multiple things at once:

test "Accounts.create_merchant_account/1 creates a merchant" do
  assert {:ok, merchant} = Accounts.create_merchant_account(%{name: "Merchant A"})

  # The description says nothing about assigning the right name
  # It's also debatable if this should even be tested in this context
  assert merchant.name == "Merchant A"

  merchant = Repo.preload(merchant, :shops)

  # The description says nothing about automagically creating a shop
  assert [%Shop{default: true}] = merchant.shops
end

# Better

test "Accounts.create_merchant_account/1 returns a success tuple with the merchant" do
  assert {:ok, merchant} = Accounts.create_merchant_account(%{name: "Merchant A"})
end

test "Accounts.create_merchant_account/1 creates a default shop" do
  assert [%Shop{default: true}] = merchant.shops
end

# Example 2
#
# Being too specific and testing the implementation instead of the outcome
test "Accounts.create_merchant/1 assigns a string timestamp + name + id to 'identifier'" do
  {:ok, merchant} = Accounts.create_merchant(%{name: "Merchant B"})
  time = merchant.inserted_at
  assert merchant.identifier == "#{time}-#{merchant.name}-#{merchant.id}"
end

# Use a lower level function to avoid testing the implementation
test "Accounts.create_merchant/1 assigns the correct identifier" do
  {:ok, merchant} = Accounts.create_merchant(%{name: "Merchant B"})
  assert Accounts.merchant_identifier(merchant) == merchant.identifier
end

# Example 3
#
# Testing side effects
test "Accounts.create_merchant/1 sends an email" do
  {:ok, merchant} = Accounts.create_merchant(%{name: "Merchant B"})
  assert however_you_would_find_out_an_email_was_sent
end

# Instead think about how you would find out as an admin user if an email has been sent
# and test the emailer separately.
test "Accounts.create_merchant/1 assigns the welcome email" do
  {:ok, merchant} = Accounts.create_merchant(%{name: "Merchant B"})

  merchant = Repo.preload(merchant, :emails)

  # Would be even better to not specify the topic like this, but use a constant. That might result
  # in over-testing though!
  assert merchant.emails == [%Email{topic: "Welcome"}]
end

test "Mailer.send_merchant_welcome/1 sends an email with the topic 'Welcome'" do
  {:ok, merchant} = Accounts.create_merchant(%{name: "Merchant B"})

  # This could look very different, depending on how the mailer works.
  assert Mailer.send_merchant_welcome(merchant) == %Mailer.Status{topic: "Welcome", status: "sent"}
end

# Example 4
#
# Testing something multiple times in higher and lower level functions

# Assuming this test calls Accounts.send_and_assign_welcome_email/1
test "Accounts.create_merchant/1 assigns the welcome email" do
  {:ok, merchant} = Accounts.create_merchant(%{name: "Merchant B"})

  merchant = Repo.preload(merchant, :emails)

  # Would be even better to not specify the topic like this, but use a constant. That might result
  # in over-testing though!
  assert merchant.emails == [%Email{topic: "Welcome"}]
end

# These tests are not needed at all, if you tested them in the calling functions
test "Accounts.send_and_assign_welcome_email/1 assigns the welcome email" do
  # ...
end