Flaky Specs

Flaky Specs

If you find a flaky spec, please submit a JIRA ticket to the Aperta project with the following format:

Type - bug

Summary - "Flaky Spec: blah blah blah"

Description - Include an error section, describing what you expected to find, and the actual error. Include the # of times this error has been seen

Label - add the label "flaky_spec"

Debugging "flaky" specs

"Flaky" is a term that is used to describe tests that work in one situation but not in another. For example, a test that passes on your dev box but not in CI.

Generally, we experience two types of flaky specs:

1. The first are flaky capybara specs. More info on these can be found here. These can be difficult to debug, and, if reasonably rare, can often just be ignored by re-running your tests.
2. The second are tests that are deterministically flaky. This page contains information on how to debug these.

Background

Because rspec runs tests in random order, there are times when an earlier test will pollute the system in such a way that later tests break. If the tests are run individually, or in a different order, the error may not be triggered.

This is usually for one of two reasons: either the earlier test inserted data into the database that was not cleaned up, or it polluted the Ruby object namespaced, e.g. by creating a temporary test class. The first type is more common.

Fortunately, we have developed a straightforward method of identified the two tests are are interacting badly: the first test that is polluting the database or object namespace and the second test that is affected.

1. Navigate to the circleci error page.

2. Click on the container that exhibited the error.

3. Copy the command line for the error (bundle exec rspec ...)

4. Copy the random seed:

5. You will now have a complete command line to run locally. You will need to change the --format option, remove the --out option and add a --seed option. When running this command, you should see the same error you saw on CI. If you can't reproduce the error, then you don't have a deterministic flaky spec.

bundle exec rspec --format documentation --seed 20704 'engines/tahi_standard_tasks/spec/features/reviewer_report_spec.rb' 'engines/tahi_standard_tasks/spec/features/register_decision_spec.rb' 'spec/features/sign_in_devise_spec.rb' 'spec/features/view_versions_spec.rb' 'engines/tahi_standard_tasks/spec/features/send_to_apex_spec.rb' 'spec/features/profile_spec.rb' 'spec/services/paper_factory_spec.rb' 'spec/models/supporting_information_file_spec.rb' 'spec/policies/nested_question_answers_policy_spec.rb' 'engines/tahi_standard_tasks/spec/features/upload_paper_spec.rb' 'spec/policies/manuscript_managers_policy_spec.rb' 'spec/services/ftp_uploader_service_spec.rb' 'engines/tahi_standard_tasks/spec/mailers/tahi_standard_tasks/reviewer_mailer_spec.rb' 'engines/plos_bio_tech_check/spec/models/changes_for_author_task_spec.rb' 'spec/services/event_stream/notifiable_spec.rb' 'spec/controllers/attachments_spec.rb' 'spec/policies/discussion_topics_policy_spec.rb' 'spec/serializers/task_serializer_spec.rb' 'spec/serializers/typesetter/competing_interests_serializer_spec.rb' 'engines/tahi_standard_tasks/spec/mailers/tahi_standard_tasks/register_decision_mailer_spec.rb' 'spec/subscribers/invitation/updated/event_stream/notify_invitee_spec.rb' 'spec/serializers/typesetter/financial_disclosure_serializer_spec.rb' 'engines/tahi_standard_tasks/spec/mailers/tahi_standard_tasks/initial_decision_mailer_spec.rb' 'spec/services/s3_form_configurator_spec.rb' 'engines/plos_bio_tech_check/spec/controllers/plos_bio_tech_check/initial_tech_check_controller_spec.rb' 'spec/models/role_spec.rb' 'spec/serializers/typesetter/funder_serializer_spec.rb' 'spec/controllers/affiliations_controller_spec.rb' 'engines/tahi_standard_tasks/spec/models/supporting_information_task_spec.rb' 'spec/controllers/ihat/jobs_controller_spec.rb' 'spec/serializers/snapshot/publishing_related_questions_task_serializer_spec.rb' 'spec/serializers/snapshot/data_availability_task_serializer_spec.rb' 'spec/controllers/countries_controller_spec.rb' 'spec/helpers/client_route_helper_spec.rb' 'spec/routing/assignments_routing_spec.rb' 'spec/helpers/mailer_helper_spec.rb' 'spec/features/mail_feedback_form_spec.rb' 'spec/services/ned_countries_spec.rb' 'spec/lib/notifier_spec.rb' ```

6. Rspec provides a --bisect option to the above command to have rspec determine the least amount of tests required to trigger your failure condition. It will run through your tests many times to determine which ones are actually actually causing your test to fail. It'll take a while (~10min) but after a while it'll spit out something like,

Round 1: searching for 86 non-failing examples (of 172) to ignore: .. (1 minute 41.23 seconds) Round 2: searching for 43 non-failing examples (of 86) to ignore: .. (1 minute 38.79 seconds) Round 3: searching for 22 non-failing examples (of 43) to ignore: . (1 minute 5 seconds) Round 4: searching for 11 non-failing examples (of 21) to ignore: .. (1 minute 21.15 seconds) Round 5: searching for 6 non-failing examples (of 11) to ignore: . (14.09 seconds) Round 6: searching for 3 non-failing examples (of 5) to ignore: .. (27.24 seconds) Round 7: searching for 2 non-failing examples (of 3) to ignore: .. (26.9 seconds) Round 8: searching for 1 non-failing example (of 2) to ignore: .. (26.9 seconds) Round 9: searching for 1 non-failing example (of 1) to ignore: . (12.92 seconds) Bisect complete! Reduced necessary non-failing examples from 172 to 1 in 7 minutes 34 seconds. The minimal reproduction command is: rspec './spec/authorizations/interface_spec.rb[1:1:4]' './spec/models/user_spec.rb[1:12:1]' --format documentation --seed 25240 ```

In this case it indicated that interface\_spec.rb is polluting
user\_spec.rb\`s state. Glad I didn't have to figure that out by
hand . . .

Attachments:

container.png (image/png)

command_line.png (image/png)

artifacts.png (image/png)

rspec_xml.png (image/png)

seed.png (image/png)

seed.png (image/png)