Skip to content

Commit

Permalink
Check for missing outcome artefacts in regression tests
Browse files Browse the repository at this point in the history
It's possible that a change in the flow and its corresponding test data may
mean that no outcome artefact is generated for one that exists in the git repo.
However, as the code stood, we would not detect this, because the assertion
diffs were only being done for the individual outcome artefacts. This change
means that we do a diff for the directory as a whole after all the response
combinations have been tried and the assertion fails if there are any
differences.

Unfortunately, because Minitest doesn't have an equivalent of RSpec's
`after(:all)`, we have to use some slightly nasty code to run our new test
at the end of all the tests for a given flow.
  • Loading branch information
floehopper committed May 27, 2015
1 parent 927286c commit 70d55fa
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 1 deletion.
6 changes: 5 additions & 1 deletion lib/smart_answer_test_helper.rb
Expand Up @@ -41,9 +41,13 @@ def read_responses_and_expected_results
YAML.load(responses_and_expected_results_yaml)
end

def path_to_outputs_for_flow
artefacts_path.join(@flow_name)
end

def save_output(responses, response)
filename = [responses.join('-'), 'html'].join('.')
path = artefacts_path.join(@flow_name)
path = path_to_outputs_for_flow
FileUtils.mkdir_p(path)
path_to_output = path.join(filename)
File.open(path_to_output, 'w') do |file|
Expand Down
8 changes: 8 additions & 0 deletions test/regression/smart_answers_regression_test.rb
Expand Up @@ -2,6 +2,9 @@
require 'gds_api/test_helpers/content_api'

class SmartAnswerResponsesAndExpectedResultsTest < ActionController::TestCase
self.i_suck_and_my_tests_are_order_dependent!
RUN_ME_LAST = 'zzzzzzzzzzz run me last'

This comment has been minimized.

Copy link
@tadast

tadast May 27, 2015

Contributor

How does this work?

Can we have assertions in the teardown method of the context?

This comment has been minimized.

Copy link
@floehopper

floehopper May 27, 2015

Author Contributor

See docs for i_suck_and_my_tests_are_order_dependent! - http://ruby-doc.com/stdlib-2.0/libdoc/minitest/unit/rdoc/MiniTest/Unit/TestCase.html#method-c-i_suck_and_my_tests_are_order_dependent-21. Essentially it means that tests are consistently run in alphabetical order. By using the "zzz" string, this should guarantee that this test runs last after all the other tests in this test case.

Putting the assertions in the context teardown won't work, because the teardown method is called after every test and not after all the tests.

This comment has been minimized.

Copy link
@tadast

tadast May 27, 2015

Contributor

I thought zzzzzzz implies sleeping :)) Thanks for the explanation, now it makes more sense why you attempted the alternative implementation :) 👍


include GdsApi::TestHelpers::ContentApi

tests SmartAnswersController
Expand Down Expand Up @@ -40,6 +43,11 @@ class SmartAnswerResponsesAndExpectedResultsTest < ActionController::TestCase
end
end
end

should "#{RUN_ME_LAST} and generate the same set of output files" do
diff_output = `git diff --stat #{smart_answer_helper.path_to_outputs_for_flow}`
assert_equal '', diff_output, "Unexpected difference in outputs for flow:"
end
end
end
end

0 comments on commit 70d55fa

Please sign in to comment.